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EXECUTIVE  SUMMARY 


Game-based  training  tools,  sometimes  called  “serious  games,”  are  becoming  more  accepted  and 
validated  as  methods  for  augmenting  and  modernizing  training  and  evaluation.  Emergency  management 
training  programs  wishing  to  include  game -based  training  in  their  curriculums  face  a  number  of  practical 
barriers;  most  importantly,  the  risk  of  failing  to  produce  a  usable  tool  at  the  end  of  a  project.  That  risk 
stems  from  several  sources:  (1)  an  unfamiliarity  with  how  to  build  serious  games;  (2)  the  fact  that 
developing  an  interactive  tool  like  a  game  is  a  more  complex,  albeit  not  necessarily  more  expensive, 
process  than  developing  other  types  of  software;  and  (3)  the  unique  requirements  for  creativity  and 
iteration  that  are  inherent  in  the  game  development  process.  These  risks  can  be  greatly  mitigated  if  the 
sponsoring  agency  selects  an  appropriate  program  structure  that  leverages  existing  talent  from  the  video 
game  industry,  supports  an  iterative  design  and  development  process,  and  draws  on  a  wider  range  of 
sources  of  creative  ideas,  involving  a  larger  number  of  contributors  in  the  process. 

In  this  document,  we  describe  seven  programmatic  structures  that  can  be  leveraged  to  create  game- 
based  training  materials.  We  describe  the  tradeoffs  of  the  different  development  approaches,  with  the  goal 
of  helping  agencies  interested  in  serious  games  select  an  appropriate  model  for  collaborating  with 
partners.  With  an  understanding  of  the  possible  collaboration  models,  we  believe  that  agencies  will  be 
able  to  explore  the  use  of  game-based  training  with  modest  budgets  and  with  a  tolerable  level  of  project 
risk. 


In  order  to  make  this  document  accessible  to  readers  who  are  new  to  serious  game  development,  we 
first  characterize  the  problem  space  in  terms  of  the  tasks  that  need  to  be  accomplished  for  designing  a 
game-based  training  tool.  Next,  we  present  the  key  organization  types  that  will  contribute  to  completing 
the  game-design  tasks,  followed  by  a  section  describing  how  the  different  organizations  match  favorably 
to  the  different  tasks.  The  organization  types  include:  government  agencies  and  training  organizations, 
national  laboratories  and  federally  funded  research  and  development  centers  (FFRDCs),  academic 
researchers  in  cognitive  science,  academic  researchers  in  game  development,  members  of  the  video  game 
industry,  general-purpose  software  developers,  and  community  participants  and  volunteers.  Then  we 
describe  a  set  of  collaboration  models  that  we  have  identified  as  pragmatic  approaches  to  developing 
game-based  training  tools.  The  collaboration  models  discussed  are:  traditional  project  sponsorship,  game 
design  competition,  project  bidding  and  development  platform,  academic  capstone  project,  commercial 
dual  release,  special  game  mode,  and  center  of  excellence.  Finally,  we  discuss  the  tradeoffs  of  each 
approach,  potential  variations  of  the  individual  development  approaches,  and  opportunities  for  hybrid 
approaches  with  the  goal  of  helping  the  reader  to  select  an  appropriate  model  for  their  program  objectives. 
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1.  INTRODUCTION 


The  use  of  interactive  digital  simulations  for  modernizing  professional  training,  often  referred  to  as 
game-based  training  or  “serious  games,”  has  attracted  a  growing  interest  in  both  the  research  and  training 
communities  [1,  2,  3,  4].  Game-based  training  for  emergency  management  professionals  can  complement 
traditional  training  methods  such  as  classroom  learning,  live  exercises,  and  on-the-job  experience.  Game- 
based  training  offers  a  number  of  appealing  benefits  to  traditional  methods,  such  as  reduced  cost,  greater 
accessibility,  flexible  scenario  creation,  and  increased  motivation  for  participation.  Game-based  training 
can  also  effectively  engage  participants  in  a  variety  of  complex  decision-making  environments  that  are 
not  easily  replicated  by  other  training  methods  [5].  These  environments  can  require  the  participants  to  use 
cognitive  and  social  skills  and  accelerate  experience  through  diverse  scenarios. 

Creating  game -based  training  tools  for  emergency  management  presents  unique  challenges,  as 
game  design  often  requires  multidisciplinary  contributors  to  develop  a  successful  product.  The  diverse  set 
of  stakeholders,  which  includes  the  sponsoring  government  agencies,  game  developers,  and  those  who 
may  benefit  from  the  training,  necessitates  collaboration  between  multiple  organizations,  which  can  lead 
to  a  challenging  management  and  development  process.  The  goal  of  this  document  is  to  focus  on  the 
practical  question  of  how  to  leverage  existing  talent  in  the  commercial,  academic,  and  federally  funded 
research  and  development  center  (FFRDC)  communities  to  help  interested  government  agencies  create 
effective  game-based  training  tools.  The  development  of  these  types  of  tools  has  much  in  common  with 
the  development  of  other  software  products,  but  there  are  some  important  differences  that  must  be  taken 
into  account.  The  creation  of  game-based  training  tools  is  still  an  emerging  area  and,  as  a  result,  these 
training  platforms  typically  involve  more  creative  design  input  and  more  rigorous  scientific  validation 
than  conventional  software  projects.  Therefore,  the  successful  development  of  game-based  training  tools 
will  likely  require  collaboration  between  several  different  contributors  bringing  different  areas  of 
expertise  to  bear. 

The  analyses  and  recommendations  in  this  document  are  supported  by  the  input  of  a  serious  games 
focus  group  held  at  the  Massachusetts  Institute  of  Technology  in  August  2015.  The  focus  group  brought 
together  working  professionals  and  thought  leaders  from  academia,  government,  national  laboratories, 
and  the  commercial  game  industry  to  discuss  practical  paths  forward  for  creating  stronger  collaborations 
in  the  area  of  game -based  training. 
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Identify  Tasks  for  Game-Based  Training  Design 


Identify  Contributing  Organizations 


Match  Organizations  to  Tasks 


Explore  Collaboration  Models 


Selecting  the  Best  Model 


Figure  1.  Document  organization. 


Figure  1  describes  the  structure  for  this  document.  We  first  characterize  the  problem  space  in  terms 
of  the  tasks  that  need  to  be  accomplished  for  designing  a  game -based  training  tool.  Next,  we  present  the 
key  organization  types  that  will  contribute  to  completing  the  tasks,  followed  by  a  section  describing  how 
the  different  organizations  match  favorably  to  the  different  tasks.  Then  we  describe  a  set  of  collaboration 
models  that  we  have  identified  as  pragmatic  approaches  to  developing  game-based  training.  Finally,  we 
discuss  the  tradeoffs  of  each  approach,  potential  variations  of  the  individual  development  approaches,  and 
opportunities  for  hybrid  approaches,  with  the  goal  of  helping  the  reader  to  select  an  appropriate  model  for 
their  program  objectives. 
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2.  TASKS  FOR  GAME-BASED  TRAINING  DESIGN 


Creation  of  a  digital  training  simulation  is  similar  to  other  software  products;  it  must  be  designed, 
developed,  validated,  and  deployed.  Along  with  those  technical  tasks,  games  require  three  additional  steps 
to  ensure  they  are  relevant  to  the  desired  training  material:  needs  analysis,  scenario  creation,  and 
curriculum  integration  must  be  performed.  Figure  2  outlines  these  seven  main  tasks  that  must  be 
completed  for  a  game-based  training  tool  to  be  created  and  have  the  desired  impact.  The  connections  and 
adjacencies  of  the  tasks  are  suggestive  of  how  the  tasks  are  interrelated  and  show  which  tasks  can  be 
performed  in  parallel.  There  will  naturally  be  many  additional  feedback  loops  in  the  progression  of  a 
program  that  are  not  depicted.  The  relative  importance  of  these  tasks  may  vary  depending  on  the  goals  of 
the  project,  but  they  will  all  be  present  in  all  projects  to  some  degree.  Next,  we  describe  each  of  the  seven 
tasks. 


Analysis  of 
Needs 


Gap  Analysis  and  Target  Skills 
Defined  Validation  Tests 


Game  Design 
&  Prototype 


Identifies 
Representative 
Scenarios  w 


Validation 


Scenario 

Creation 


Development 


Tailored  to 
Course  Goals 


Validation  Continues  as  Student 
Feedback  and  Data  is  Collected 


Curriculum 

Integration 


Development  is  Done  with  Hosting 
Lifespan  and  Technology  in  Mind 


Game  is  Adapted  to  Different 
Curriculums  and  Regimens 


Hosting  and 
Distribution 


Hosting  May  Continue  for 
Skill  Maintenance  or 
Popular  Training  Programs 


Figure  2.  Seven  tasks  involved  in  the  creation  of  effective  game-based  training  materials. 


•  Analysis  of  Needs:  The  training  need  and  programmatic  objectives  are  articulated  so  they  can 
guide  the  appropriate  design  or  selection  of  a  game.  For  example,  what  skills  or  decisions 
should  be  targeted,  what  scenarios  are  likely  or  of  particular  concern,  and  what  type  of  decision 
maker  is  going  to  be  the  target  beneficiary  of  the  game? 

In  this  step  it  is  also  important  to  consider  whether  a  game  is  appropriate  for  the  desired  goals 
of  the  training.  Games  are  an  effective  method  for  engaging  the  participant  in  a  dynamic 
decision-making  environment  and  allowing  the  participant  to  understand  the  impact  of  using 
different  strategies.  Games  are  also  a  powerful  tool  for  the  training  staff,  enabling  the  capability 
to  generate  randomized  modifications  to  the  game  scenario  to  ensure  students  are  learning  the 
desired  material  and  not  just  memorizing  the  solution.  Examples  of  the  types  of  decisions  and 
skills  we  have  investigated  for  game -based  training  are  documented  in  another  report  [24]. 
Game-based  training  is  not  always  the  best  tool,  and  conventional  training  and  exercises  should 
still  be  employed  to  meet  needs  that  they  are  well  suited  for,  such  as  rehearsing  procedures, 
learning  factual  information,  and  building  connections  between  organizations  via  social 
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interaction.  The  Analysis  of  Needs  stage  should  identify  what  type  of  training  material  will  be 
best  suited  to  meet  existing  gaps. 

•  Game  Design  and  Prototype:  A  suitable  game  is  designed  or  adapted  from  a  prior  design 
template.  A  prototype  of  the  design  should  be  created  to  get  feedback  from  the  target  user  base 
and  to  calibrate  scoring  rules  and  scenario  difficulty.  At  this  stage,  the  design  could  be  as 
simple  as  a  description  of  the  rules  and  mechanics  but  should  offer  some  ability,  even  if  on 
paper,  to  be  played  and  tested.  The  game  design  might  need  several  iterations  to  match  it  to  the 
target  audience’s  needs. 

•  Development:  A  full  working  version  of  the  game  is  created.  This  version  will  be  based  on  the 
approved  prototype,  but  implemented  in  software,  with  improved  graphics,  stability,  additional 
gameplay  content,  and  instrumented  data  tracking.  The  developed  game  will  include  several 
scenarios  to  represent  the  range  of  possibilities,  but  the  main  focus  will  remain  on  the  core 
engine  for  the  game  mechanics. 

•  Scenario  Creation:  A  larger  suite  of  scenarios  is  created.  A  tool  that  can  automatically  or 
manually  generate  new  scenarios  from  seed  data  may  also  be  created.  Different  scenarios  can 
target  different  situations  (e.g.,  a  high  surge  hurricane  vs.  a  high  wind  hurricane),  different  user 
bases  (e.g.,  a  weather  analyst  vs.  an  emergency  manager),  or  different  threats  (e.g.,  a  false 
positive  vs.  a  true  threat).  For  certain  games,  a  scenario  editor  may  also  be  part  of  this  process, 
allowing  end  users  to  create  novel  scenarios  from  their  own  experiences. 

•  Validation:  A  formal  analysis  of  the  efficacy  of  the  game  involves  trials  with  human  subjects. 
The  level  of  rigor  may  vary  depending  on  the  project,  but  some  level  of  confirmation  will  be 
important  to  any  project.  Validation  may  be  an  informal  assessment  by  experts,  a  self- 
assessment  survey  of  users,  a  formal  performance  evaluation  such  as  a  test,  a  material  retention 
study  conducted  over  time,  or  any  combination  of  these  techniques. 

•  Curriculum  Integration:  The  game  is  integrated  with  appropriate  complementary  training 
materials.  For  example,  the  game  might  be  paired  with  (1)  a  preexisting  appropriate  training 
course  as  is,  (2)  a  training  course  where  supplemental  written  material  might  need  to  be  revised, 
or  (3)  a  program  for  skill  maintenance  might  need  to  be  created  to  encourage  users  to  keep 
using  the  game  over  time. 

•  Hosting  and  Distribution:  The  game  is  hosted  so  that  it  can  be  accessed  by  users  -  both 
trainers  and  students.  Depending  on  the  program  objectives  of  the  game,  it  might  be  available 
only  with  a  matching  course  registration,  to  all  professionals  in  the  field,  or  to  the  general 
public.  It  might  need  to  be  available  all  the  time  for  skill  maintenance  or  only  on  occasion  in 
support  of  periodic  training  programs. 

In  conclusion,  many  of  these  tasks  will  be  completed  within  the  scope  of  the  initial  program. 
However,  some  tasks  may  exist  past  the  end  of  the  initial  program.  The  validation  of  the  game  may 
evolve  into  a  continuous  process,  revising  game  content  based  on  user  feedback,  while  the  evolution  of 
certain  curricula  may  require  the  game  to  be  modified. 
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3.  CONTRIBUTING  ORGANIZATIONS 


For  each  of  the  tasks  outlined  in  the  previous  section,  there  are  one  or  more  roles  to  play;  one 
organization  will  be  the  lead  for  accomplishment  of  the  task  but  may  require  support  from  other 
contributors  with  complementary  strengths.  It  is  unlikely  that  any  one  organization  would  be  able  to 
effectively  lead  or  possibly  even  participate  in  every  task.  Furthermore,  it  would  be  undesirable  to  have 
one  organization  fill  all  roles,  as  that  would  discourage  an  iterative  design  process,  limit  the  range  of 
creative  inputs,  and  introduce  bias  in  the  validation.  However,  a  single  organization  could  play  a  role  on 
several  different  tasks.  In  this  document,  we  advocate  for  a  small  set  of  well-selected  collaborators,  as  it  is 
likely  to  be  more  efficient  and  focused  than  a  long  list  of  partners. 


Academic  Researchers: 
Cognitive  Science 


Academic  Researchers: 
Game  Development 


National  Laboratories 
and  FFRDCs 


Government  Agencies 
and  Training 
Organizations 


Community  Members 
and  Volunteers 


General  Purpose 
Software  Developers 


Video  Game  Industry 


Figure  3.  Potential  contributors  to  the  development  of  game-based  training. 


Figure  3  shows  seven  categories  of  organizations  that  could  fill  key  roles  in  the  development  of 
game-based  training  materials.  Note  that  not  all  types  of  organizations  will  fill  a  role  in  all  projects,  and 
only  a  select  subset  should  be  leveraged  for  any  particular  project. 

•  Government  Agencies  and  Training  Organizations.  Government  agencies  might  sponsor 
research,  create  training  materials,  or  wish  to  be  users  of  modernized  training  programs.  They 
can  provide  domain  knowledge  and  guide  the  requirements  for  the  game.  This  category 
includes  agencies  such  as  the  Department  of  Homeland  Security  (DHS)  Science  and 
Technology  Directorate  (S&T),  the  Federal  Emergency  Management  Agency  (FEMA),  and 
local  emergency  manager  training  divisions. 

Government  agencies  traditionally  collaborate  with  contracted  developers,  national  laboratories, 
and  academic  centers  of  excellence.  However,  they  may  need  to  forge  closer  connections  with 
members  of  the  user  community,  such  as  first  responders  and  emergency  managers,  to  allow 
those  communities  to  be  more  actively  involved  in  the  creation  and  maintenance  of  the  training 
materials. 
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•  National  Laboratories  and  FFRDCs.  National  laboratories  and  FFRDCs  serve  as  a  bridge 
between  government  agencies  with  technical  needs  and  commercial  contractors  who  can  supply 
products  to  meet  those  needs.  They  also  serve  as  unbiased  advisors  of  the  shape  of  a  project, 
helping  to  define  the  requirements,  perform  research,  and  build  prototypes  to  reduce  risk  and 
provide  support  to  government  programs. 

National  Laboratories  traditionally  collaborate  with  the  government  (via  sponsorship),  industrial 
contractors  (via  tech  transfers),  and  academic  researchers  (via  project  collaboration).  To  support 
game-based  training,  they  may  need  to  forge  better  connections  with  the  video  game  industry 
and  new  types  of  academic  departments. 

•  Academic  Researchers:  Cognitive  Science.  Academic  researchers  in  the  areas  of  cognitive 
science,  psychology,  and  human  factors  can  supply  expertise  on  how  to  understand  user 
behavior  and  needs.  They  can  help  to  create  and  validate  concepts  that  support  human-intensive 
tasks,  and  they  can  bring  recent  research  to  bear  on  a  practical  problem. 

Academic  research  teams  traditionally  collaborate  with  government  agencies  via  grants  and 
center  of  excellence  programs,  as  well  as  with  national  laboratories  and  other  academic 
disciplines.  However,  those  collaborations  are  typically  isolated  from  each  other,  and  may  need 
to  become  part  of  a  unified  effort  to  support  the  development  of  game -based  training. 

•  Academic  Researchers:  Game  Development.  Academic  game  development  programs  often 
focus  on  preparing  students  to  work  in  the  games  industry.  The  professors  have  theoretical  and 
practical  knowledge  of  a  wide  range  of  games  and  how  those  games  can  target  different 
audiences,  and  the  students  represent  young  talent  in  the  areas  of  game  design,  software 
engineering,  and  the  practical  application  of  artistic  skills. 

Game  degree  programs  often  collaborate  with  the  games  industry  (to  help  their  students  find 
employment)  and  other  academics  (to  further  their  own  research).  They  may  need  to  become 
more  involved  with  national  laboratories  to  better  understand  the  serious  games  problem  space 
and  to  leverage  government  training  needs  as  an  asset  for  providing  their  students  with  practical 
game  development  experience. 

•  Video  Game  Industry.  The  video  game  industry  has  developed  a  host  of  tools  and  techniques 
for  identifying,  capturing,  and  retaining  audiences.  Modern  video  games  often  utilize  deep 
strategic  decision-making  experiences,  compelling  narrative  story  lines,  and  game  mechanics  to 
make  gameplay  accessible  to  new  players,  as  well  as  visually  and  auditory  stimulating 
environments  to  create  an  engaging  experience  for  players.  Many  independent  game  developers 
(often  referred  to  as  “indie”  developers)  have  novel  and  creative  ideas  for  what  games  can 
accomplish. 

The  games  industry  often  collaborates  with  academic  programs  (supplying  new  hires)  and  non¬ 
game  software  programs  (providing  technical  infrastructure),  and  they  are  very  effective  at 
involving  their  users  and  players  in  the  creation  of  in-game  content  (via  custom  levels,  strategy 
forums,  and  fan-written  narrative  backstories  for  characters).  While  many  members  of  the 
games  industry  are  interested  in  having  an  impact  in  the  serious  games  space,  they  need 
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assistance  from  the  other  organizations  listed  in  this  section  in  order  to  forge  better  collaboration 
with  government  entities. 

•  General-Purpose  Software  Developers.  There  are  a  host  of  companies  providing  general- 
purpose  software  development  services.  Many  of  them  have  experience  collaborating  with 
government  projects,  but  they  are  unlikely  to  bring  to  bear  talent  specific  to  the  creation  of 
games  and  modernized  training  environments. 

General-purpose  software  contractors  might  be  leveraged  in  limited  ways  in  such  efforts,  but  in 
general,  government  sponsors  will  need  to  look  beyond  their  traditional  contractors  to  meet 
their  needs  in  the  area  of  modem  training  games. 

•  Community  Participants  and  Volunteers.  One  of  the  great  successes  of  many  entertainment 
games  is  the  active  involvement  of  paying  players.  The  experiences  created  by  successful  video 
games  are  so  compelling  that  players  volunteer  their  time  and  talent  to  enrich  the  world  with 
custom  levels,  strategic  fomms,  and  narrative  backstories.  A  similar  level  of  engagement  with 
relevant  participants  in  emergency  management  could  have  a  great  impact  on  the  field  of 
emergency  management  game-based  training.  Community  participants  and  volunteers  in  this 
domain  could  include  professionals  who  would  be  the  planned  users  of  the  game,  general 
gamers  who  are  intrigued  by  disaster  response,  or  members  of  the  public  who  want  to  be 
involved  in  community  preparation. 


Figure  4.  Traditional  (solid)  and  emerging  (dashed)  collaborations  between  potential  contributors  to  a  game-based 
training  project. 
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Figure  4  summarizes  how  these  groups  have  traditionally  collaborated  and  what  potential  and 
emerging  collaborations  would  support  the  creation  of  game -based  training  materials. 

•  Government  agencies  and  national  laboratories  have  a  history  of  working  with  each  other,  with 
academic  researcher  teams,  and  with  general-purpose  software  development  companies. 
However,  neither  has  a  history  of  working  closely  with  game  development  academics  or  the 
video  game  industry,  both  of  which  can  be  strong  partners  in  the  creation  of  game-based 
training  materials.  National  laboratories  are  likely  to  be  a  portal  for  bridging  that  gap,  which 
calls  for  building  stronger  ties  (dashed  lines  on  the  right  side  of  the  diagram)  to  those  potential 
partners. 

•  Academic  game  degree  programs  and  commercial  video  game  companies  are  not  entirely 
disconnected  from  government  agencies  at  present.  They  already  have  indirect  ties  that  can  be 
leveraged  to  help  forge  more  direct  ties.  As  indicated  in  Figure  4,  video  game  companies 
already  work  closely  with  general-purpose  software  developers,  and  academic  game  degree 
programs  are  often  closely  linked  to  academic  cognitive  science  departments.  Creating  more 
direct  links  (indicated  by  the  dashed  lines)  will  be  important,  and  it  is  a  goal  that  is  achievable 
in  the  near  term. 

•  The  video  game  industry  also  has  a  successful  record  of  creating  environments  where  their  fans 
contribute  creative  content  using  their  own  time  (as  shown  by  the  solid  line  at  the  bottom  of 
Figure  4),  which  is  a  level  of  community  engagement  that  government  training  organizations 
could  benefit  from  emulating  (as  shown  by  the  far  left  dashed  line  in  Figure  4).  Engaging  game- 
based  training  with  tools  to  support  user-created  content  (e.g.,  custom  scenarios  and  situations 
for  an  existing  training  game)  could  help  to  leverage  existing  talent  in  the  emergency  response 
community  toward  the  creation  of  training  materials. 
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4.  MATCHING  ORGANIZATIONS  TO  TASKS 


The  key  tasks  and  organizations  described  in  Sections  2  and  3  can  be  matched  together  in  a  range  of 
combinations.  The  tasks  will  be  completed  more  effectively  if  there  is  a  well-structured  collaboration 
between  the  organizations  involved.  Figure  5  shows  which  organizations  are  likely  to  be  effective  at 
filling  each  of  the  tasks. 

Rather  than  rate  the  appropriateness  on  a  traditional  strong/moderate/weak  scale,  we  have  opted  to 
categorize  them  on  the  role  they  would  likely  play  in  a  collaboration  -  leading  or  supporting  the  task. 
Note  that  “supporter”  is  not  considered  to  be  a  weaker  role  than  “leader,”  but  merely  a  different  one.  The 
leader  may  provide  the  primary  guidance  and  strategic  vision,  while  the  supporter  may  provide  the  bulk 
of  the  labor  and  content  to  accomplish  that  vision. 

Along  with  the  seven  roles  described  earlier,  we  have  included  the  role  of  “process  coordinator.” 
Coordinating  diverse  communities  such  as  the  games  industry  and  government  training  organizations  is 
entirely  possible,  but  it  will  require  thought  and  care  in  how  to  organize  and  sustain  healthy  and  mutually 
beneficial  collaborations.  In  more  traditional  fields,  the  government  sponsor  or  an  academic  expert  might 
be  sufficient  to  lead  that  effort.  In  the  area  of  serious  games,  those  actors  have  an  important  part  to  play, 
but  the  leadership  should  come  from  a  third  party  with  experience  in  both  the  application  domain  and  in 
the  field  of  serious  game  design. 
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Figure  5.  The  tasks  each  type  of  organization  is  suited  to  lead  and  support  in  the  development  of  digital  training 
simulations. 
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Identifying  a  set  of  actors  who  fill  the  necessary  gaps  is  a  start,  but  it  is  not  sufficient  to  build  a 
successful  collaboration.  For  example,  Government  Agencies  and  Training  Organizations,  National 
Laboratories  and  FFRDCs,  and  the  Video  Game  Industry  (the  first,  second,  and  fifth  rows)  can  either  lead 
or  support  all  the  required  tasks  (represented  by  the  columns),  but  identifying  those  contributors  is  only 
part  of  building  an  effective  collaboration.  It  is  also  important  to  understand  the  structure  of  their 
collaboration,  and  ensure  that  the  structure  leverages  each  group’s  strengths  and  accommodates  their 
weaknesses. 

One  of  the  main  obstacles  is  that  many  of  these  organizations  have  very  different  modes  of 
operation  and  different  goals;  as  a  result,  they  are  all  not  familiar  with  how  the  other  types  of 
organizations  operate.  For  example,  building  collaboration  of  independent  game  developers  operating 
with  little  or  no  management  overhead  and  shoestring  budgets  with  a  government  organization  with 
bureaucratic  processes  and  low  risk  tolerance  is  not  straightforward.  Those  organizations  have  a  lot  to 
offer  each  other,  but  they  traditionally  operate  in  very  different  fashions.  Forging  an  alliance  between 
them  will  require  thinking  about  new  or  modified  models  for  collaboration. 

Another  challenge  in  creating  appropriate  collaboration  models  is  that  game  design  is  iterative.  Like 
other  user-facing  products,  creating  an  effective  game  (be  it  for  entertainment  or  training  purposes)  is 
fundamentally  an  iterative  process  and  rarely  follows  a  linear  design  path.  If  forced  into  a  traditional  sole- 
source  sponsorship  model,  game-based  training  tools  are  likely  to  suffer  -  they  will  be  more  likely  to  fail 
outright  or  to  end  up  as  successes  only  on  paper,  with  limited  real  impact.  However,  with  a  model 
supporting  iteration,  those  risks  can  be  greatly  mitigated.  The  program  structure  of  game  design  requires 
support  for  multiple  contributors,  collaboration,  feedback  loops,  and  an  acceptance  that  the  shape  of  the 
final  product  may  not  be  known  up  front.  These  are  properties  that  can  be  unfamiliar  to  government 
sponsors,  and  may  require  some  acclimation  on  the  part  of  the  sponsoring  agency  overseeing  the  work. 
However,  iterative  design  is  widely  accepted  in  industry  and  there  is  precedent  for  government  agencies 
successfully  using  “agile  methods”  to  manage  projects  [6].  That  trend  extends  far  beyond  just  the  creation 
of  games,  and  the  collaboration  models  we  describe  in  this  document  may  also  apply  more  broadly  than 
just  to  game -based  training. 

While  game  design  has  inherent  uncertainty  due  to  the  reasons  mentioned,  there  are  models  for 
collaboration  that  can  reduce  the  risk.  Examples  of  those  models  include  leveraging  experts  in  the  field, 
using  ongoing  collaborative  feedback  mechanisms,  and  spreading  risk  over  a  wide  range  of  contributors. 
While  these  models  may  not  follow  the  traditional  path,  a  closer  look  shows  that  most  of  them  are 
analogous  to  conventional  project  structures,  with  adaptations  for  some  of  the  unique  properties  of  game 
design.  For  example,  a  game  competition  is  similar  to  a  Defense  Advanced  Research  Projects  Agency 
(DARPA)  challenge  or  competitive  bidding  process,  and  a  capstone  project  is  similar  to  engaging  a  center 
of  excellence  or  funding  an  academic  conference. 
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5.  COLLABORATION  MODELS 


This  section  describes  how  to  foster,  motivate,  and  direct  the  collaboration  between  the  diverse  set 
of  contributors  to  reduce  programmatic  risk  and  increase  the  ultimate  impact  of  the  resulting  product.  It  is 
important  to  understand  that  there  is  more  to  a  collaboration  model  than  just  identifying  who  does  what 
job.  The  structure  of  the  collaboration,  namely,  how  information  is  shared  and  how  work  is  distributed 
amongst  the  collaborators,  is  a  critical  part  of  the  equation,  and  such  structures  must  be  selected  to  suit 
the  project  goals  and  types  of  collaborators  involved. 

We  have  assessed  seven  collaboration  models,  each  of  which  has  potential  variations.  Some  of 
these  models  can  be  combined  to  create  hybrid  approaches  in  order  to  mix  and  match  models  for  different 
phases  of  the  project;  as  a  result,  they  should  not  be  considered  mutually  exclusive  options.  We  will 
examine  the  following  models: 

A.  Traditional  Project  Sponsorship 

B.  Game  Design  Competition 

C.  Project  Bidding  and  Development  Platform 

D.  Academic  Capstone  Project 

E.  Commercial  Dual  Release 

F.  Special  Game  Mode 

G.  Center  of  Excellence 

5.1  MODEL  A.  TRADITIONAL  PROJECT  SPONSORSHIP 

A  traditional  project  sponsorship  model  closely  emulates  a  familiar  path  for  many  government 
agencies,  involving  a  mostly  linear  development  cycle  with  a  carefully  selected  sole  performer 
completing  each  task. 
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Figure  6.  An  example  implementation  of  the  Traditional  Project  Sponsorship  model 


Figure  6  shows  an  example  implementation  of  this  model.  The  sponsoring  agency  and  relevant 
subject  matter  experts  (SMEs)  support  a  national  laboratory  to  assess  the  training  needs.  The  laboratory 
then  creates  a  game -based  training  prototype  to  meet  that  need.  The  prototype  step  reduces  risk  by 
allowing  for  lightweight  design  iterations  and  several  rounds  of  feedback  from  SMEs  to  refine  the  game 
design  prior  to  a  larger  development  effort.  The  full  development  effort  is  then  handed  off  to  a  single¬ 
source  performer  from  private  industry  (e.g.,  a  software  development  company  from  the  video  game 
industry).  That  company  creates  the  production  quality  game  and  then  supports  a  government  training 
organization  in  adding  scenarios  and  other  domain-specific  content  into  that  framework.  Academic 
researchers  validate  the  effectiveness  of  the  end  product  and,  based  on  their  findings,  support  effective 
integration  into  a  training  curriculum.  In  the  long  term,  the  government  training  organization  hosts  and 
distributes  the  game  as  part  of  a  training  or  assessment  program. 

This  model  is  very  close  to  typical  government  program  structures,  with  some  important 
adjustments  that  make  it  effective  for  game-based  training.  SMEs  are  not  just  involved  in  the 
identification  of  the  training  goal  or  need;  they  are  also  actively  included  in  the  prototyping  and 
development  efforts,  serving  as  play  testers  in  a  highly  iterative  process.  The  sponsoring  agency  will  need 
to  ensure  that  SMEs  are  made  available  to  the  performers  throughout  the  process.  There  is  an  explicit 
validation  step  to  confirm  that  the  final  product  fills  the  needs  identified  earlier  in  the  process.  The  leader 
of  the  validation  step  should  be  separate  from  the  leaders  of  the  design  and  development  steps,  to  remove 
potential  bias.  In  the  depicted  version,  that  means  using  an  academic  research  team  separate  from  the 
national  laboratory  involved  earlier  in  the  process. 
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Figure  7.  The  roles  each  contributor  might  play  under  a  Traditional  Project  Sponsorship  type  of  collaboration. 


5.1.1  Strengths  and  Weaknesses 

A  traditional  project  sponsorship  model  works  best  when  the  techniques  and  technologies  involved 
are  already  well  understood,  so  that  a  design  can  be  selected  and  frozen  up  front.  The  adaptations 
described  earlier  help  to  inject  some  of  the  iterative  development  cycles  necessary  to  game  development. 
A  single  performer  following  a  requirements  document  as  a  legal  contract  will  reliably  produce  a 
functioning  product  that  meets  the  design,  but  may  not  actually  meet  the  original  need.  As  the  field  of 
game-based  training  matures,  this  approach  may  become  more  viable,  but  in  the  current  state,  it  should  be 
used  sparingly,  and  alternatives  should  be  considered,  if  viable,  in  the  political  climate  of  the  sponsoring 
agency. 

5.2  MODEL  B.  GAME  DESIGN  COMPETITION 

A  game  design  competition  is,  at  its  heart,  a  kind  of  broad  agency  announcement  (BAA)  process 
where  the  participants  demonstrate  their  abilities  and  ideas  as  part  of  the  bid.  The  project  then  benefits  not 
only  from  seeing  a  wide  range  of  ideas,  but  also  from  having  more  information  about  the  abilities  of  the 
candidate  performers.  The  structure  may  be  familiar  to  both  government  sponsors  (e.g.,  DARPA  Urban 
Challenge,  DARPA  Robotics  Challenge)  and  game  developers  (e.g.,  Global  Game  Jam,  Game  Jam 
Central),  and  thus  be  effective  at  bringing  those  communities  together. 
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Figure  8.  An  example  implementation  of  the  Game  Design  Competition  model. 


Figure  8  illustrates  a  sample  workflow  for  a  game  design  competition.  The  process  begins  with  an 
analysis  of  needs  led  by  the  interested  government  training  agency.  Rather  than  making  an  up-front 
decision  on  a  single  design  to  meet  the  identified  needs,  the  statement  of  need  is  provided  as  a  challenge 
for  a  game  competition.  Participants  submit  ideas  and  quick  prototypes  for  games  to  meet  those  needs, 
and  the  best  ones  are  selected  for  refinement  over  two  to  three  rounds.  As  the  contest  proceeds,  the  rounds 
shift  from  being  more  about  finding  promising  ideas  to  developing  those  ideas  into  working  games.  The 
finalist  games  are  then  augmented  with  scenarios  and  domain  content  provided  by  the  training  agencies 
that  wrote  the  original  statements  of  needs.  Up  to  this  point,  a  national  laboratory  phrases  the  analysis  of 
needs  in  terms  of  a  game  challenge,  judges  the  contests,  provides  guidance  to  teams,  and  helps  adapt 
scenario  needs  to  developed  games.  The  national  laboratory  then  takes  a  more  active  role  in  validating  the 
resulting  game -based  training  materials,  before  those  materials  are  integrated  into  a  training  curriculum 
and  hosted  for  ongoing  use. 

The  process  has  many  similarities  to  a  traditional  BAA  bidding  process.  It  is  common  for 
companies  bidding  for  a  high-stakes,  software -based  BAA  to  demonstrate  their  ability  in  the  area  with  a 
simple  mock-up  or  prototype  of  the  capability  required,  both  to  win  the  bid  and  to  help  the  government 
sponsor  understand  what  capabilities  already  exist.  For  many  independent  developers,  a  contest  with  $50k 
in  prize  money  would  qualify  as  a  high-stakes  project  and  gamer  a  similar  level  of  investment  and 
interest. 

The  Game  Design  Competition  model  differs  from  a  traditional  BAA  by  including  the  initial 
prototype  into  the  bidding  process,  and  offers  consolation  prizes  for  submissions  that  are  not  ultimately 
selected.  However,  the  end  result  is  very  similar  to  a  BAA  and  would  produce  comparable  benefits  for  the 
government  sponsor. 
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Figure  9.  The  roles  each  contributor  might  play  under  a  Game  Design  Competition  type  of  collaboration. 


5.2.1  Strengths  and  Weaknesses 

This  model  is  particularly  appealing  in  the  area  of  game-based  training,  since  there  is  a  large 
community  of  independent  game  developers  with  strong  creative  and  technical  skills,  available 
inexpensively,  and  often  strongly  motivated  to  make  a  positive  impact  on  the  world.  This  community  is 
very  familiar  with  the  notion  of  a  game  jam  (a  short  time-frame  game  design  competition),  so  such  a 
structure  could  be  an  inexpensive  way  to  leverage  an  experienced  talent  pool  -  a  prize  pool  of  $5 Ok  or 
less  would  attract  top  minds  from  the  community  and  provide  a  large  pool  of  ideas  to  select  from,  along 
with  the  talent  to  develop  and  mature  those  ideas.  When  overseen  by  an  experienced  national  laboratory, 
the  model  can  also  be  comfortable  for  the  sponsoring  government  agency.  The  laboratory  provides 
cohesion  to  the  effort,  keeps  the  participants  focused  on  the  project  goals,  helps  the  independent 
participants  manage  government  bureaucracy,  and  serves  as  a  single  point  of  contact  for  the  sponsoring 
agency. 

Nevertheless,  this  model  will  feel  unfamiliar  to  many  sponsoring  agencies.  As  with  other  iterative 
development  processes,  sponsoring  agencies  will  need  to  acclimate  to  the  notion  of  not  knowing  what  the 
final  product  will  look  like  even  while  development  is  underway,  and  instead  trust  that  many  iterations 
and  expert  feedback  will  keep  the  program  on  track. 

5.3  MODEL  C.  PROJECT  BIDDING  AND  DEVELOPMENT  PLATFORM 

A  bidding  platform  aims  to  reduce  the  overhead  of  funding  many  small  programs  in  an  area  by 
sharing  overhead  between  those  platforms.  It  would  be  appropriate  when  (or  if)  game-based  training 
becomes  more  widespread  and  building  infrastructure  to  support  their  development  is  cost  effective. 
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Figure  10.  An  example  implementation  of  the  Project  Bidding  and  Development  Platform  model. 


A  possible  work  flow  of  this  model  is  shown  in  Figure  10.  An  initial  survey  identifies 
commonalities  among  the  typical  needs  of  agencies  with  training  needs  suitable  to  game -based  training.  A 
general-purpose  software  contractor,  with  guidance  from  a  national  laboratory  experienced  with  serious 
games,  develops  a  platform  providing  tools,  support,  and  bidding  functionality.  Then  individual 
government  agencies  can  advertise  their  needs  on  the  platform  and  receive  bids  from  developers  in  the 
video  game  industry.  Bidders  would  be  able  to  leverage  provided  common  libraries  such  as  models  for 
how  populations  move  during  a  disaster,  weather  models,  game  design  templates,  and  art  assets  focused 
on  emergency  response.  Scenarios  for  the  games  could  also  be  crowdsourced  via  the  platform,  allowing 
professional  first  responders  and  emergency  managers  to  provide  content  for  existing  game  templates  and 
designs.  The  resulting  games,  with  their  domain-specific  scenarios,  could  be  validated  and  integrated  into 
training  programs.  The  platform  would  also  provide  basic  hosting  services,  so  that  individual  government 
training  organizations  would  not  need  those  skills  in-house. 
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Figure  11.  The  roles  each  contributor  might  play  under  a  Project  Bidding  and  Development  Platform  type  of 
collaboration. 


5.3.1  Strengths  and  Weaknesses 

A  project  bidding  platform  can  be  an  effective  tool  for  lowering  the  overhead  cost  for  funding  small 
projects  in  a  field.  Tracking  ratings  and  reputation  for  performers  would  reduce  risk  of  the  government 
agency  requesting  bids,  and  allow  them  to  access  talented,  low-cost  freelance  labor.  However,  this  model 
requires  critical  mass  of  supply  and  demand  for  game-based  training  to  justify  the  cost  of  maintaining  the 
platform  and  to  attract  enough  attention  from  the  games  industry  to  quickly  meet  calls  for  bids.  However, 
once  sufficient  interest  from  both  sides  exists,  a  bidding  platform  could  greatly  reduce  the  time  and  dollar 
cost  of  creating  training  materials. 

5.4  MODEL  D.  ACADEMIC  CAPSTONE  PROJECT 

Academic  capstone  projects  are  common  in  engineering  programs  as  a  means  of  giving  students 
experience  working  as  a  team  on  an  end-to-end  project.  The  general  goal  of  capstone  projects  is  to 
provide  practical  experience  to  students  prior  to  graduation.  As  demonstrated  by  the  Beaver  Works 
program  [7],  students  can  also  be  leveraged  to  meet  government  needs,  which  assists  the  participating 
agency  and  provides  the  students  a  more  realistic  and  fulfilling  educational  experience.  This  model  could 
be  most  applicable  to  universities  with  academic  game  degree  programs  where  students  are  trained  to 
design  and  develop  video  games. 
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Figure  12.  An  example  implementation  of  the  Academic  Capstone  Project  model. 


One  way  to  organize  such  a  program  is  shown  in  Figure  12.  A  national  laboratory  leads  the 
assessment  of  needs,  and  then  continues  to  coordinate  and  support  the  subsequent  steps.  The  needs 
assessment  is  provided  as  a  capstone  project  goal  to  senior-level  students  in  an  academic  game  degree 
program,  who  then  prototype  the  game  with  feedback  from  SMEs.  Depending  on  the  nature  of  the 
program,  the  capstone  students  might  also  be  involved  in  the  analysis  of  needs  to  help  them  build  domain 
analysis  skills  and  better  understand  the  needs  of  their  users.  The  final  projects  are  then  handed  to  a 
professional  development  company  for  production  quality  development  and  to  support  the  addition  of 
custom  scenarios.  The  national  laboratory  completes  its  role  by  validating  the  final  product  in  preparation 
for  curriculum  integration  and  hosting. 
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Figure  13.  The  roles  each  contributor  might  play  under  an  Academic  Capstone  Project  type  of  collaboration. 


5.4.1  Strengths  and  Weaknesses 

Leveraging  students  in  this  manner  is  a  very  inexpensive  way  to  access  top  talent  and  fresh  ideas.  It 
also  serves  to  direct  the  attention  of  young  professionals  toward  the  needs  and  issues  of  the  participating 
government  agency.  This  model  may  be  best  used  as  a  precursor  and  supplement  to  another  model  -  in 
the  diagrams  above,  it  can  be  seen  pairing  with  a  traditional  project  sponsorship  (Model  A). 

5.5  MODEL  E.  COMMERCIAL  DUAL  RELEASE 

A  commercial  dual  release  is  when  a  private  company  receives  partial  funding  for  creating  a  product 
family  that  includes  both  a  commercial  for-profit  version  and  a  version  delivered  to  the  government 
sponsor  to  be  used  as  a  training  tool.  This  technique  that  has  been  leveraged  by  past  Department  of 
Defense  (DoD)  programs  [8,  9]. 
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Figure  14.  An  example  implementation  of  the  Commercial  Dual  Release  model. 


Figure  14  shows  how  such  a  collaboration  might  unfold.  A  national  laboratory  helps  identify  an 
area  of  need  that  aligns  with  popular  video  game  styles  and  helps  guide  a  commercial  company  in 
building  such  a  game.  The  private  company  designs  and  develops  the  game  and  helps  the  government 
training  organization  build  scenarios  for  it.  The  government  then  takes  one  version  of  the  game  as 
government  off-the-shelf  (GOTS)  software  and  validates,  integrates,  and  hosts  it.  The  commercial  partner 
takes  a  different  version  of  the  same  game  and  releases  it  as  a  commercial  product. 
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Figure  15.  The  roles  each  contributor  might  play  under  a  Commercial  Dual  Release  type  of  collaboration. 
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5.5.1  Strengths  and  Weaknesses 

When  this  model  works,  it  can  be  a  big  win  for  both  parties.  The  government  gets  training  materials 
at  a  greatly  reduced  price,  and  the  commercial  partner  gains  a  long-term  revenue  stream  with  little  up¬ 
front  investment.  However,  there  are  two  important  challenges  in  making  the  model  work. 

Challenge  1:  Alignment  of  Interests 

It  is  hard  to  find  opportunities  where  training  and  entertainment  interests  are  sufficiently  aligned. 
The  successes  in  the  DoD  were  based  on  squad-level  tactics  where  realism  of  the  commercial  product  was 
a  selling  point,  and  the  general  domain  (modem  tactical  shooters)  had  a  history  of  commercial  viability. 
Making  such  a  model  work  for  a  non-militarized  domain  such  as  emergency  management  or  first 
response  is  a  challenge,  but  not  an  impossible  one.  For  example,  the  recent  popularity  of  the  “survival” 
genre  of  entertainment  game  may  be  a  point  of  common  interest;  players  attempt  to  survive  as  long  as 
possible  in  a  hostile  environment  by  foraging  and  evading  threats,  rather  than  violent  confrontation  of  the 
threats.  Such  a  game  might  serve  to  train  the  public  or  leam  how  the  public  responds  to  different 
situations. 

Another  variation  of  this  approach  is  to  create  a  shared  game  engine  rather  than  a  shared  game.  The 
partners  share  the  costs  of  developing  the  core  technology,  but  then  diverge  in  which  scenarios  and 
settings  are  recreated  with  that  technology.  Many  of  the  skills  that  can  be  effectively  trained  with  game- 
based  techniques  are  strategic  skills  and  soft  skills.  Those  skills  are  commonly  part  of  successful 
entertainment  games,  and  an  effective  training  tool  might  become  a  commercially  viable  product  simply 
by  recasting  it  in  a  different  setting.  For  example,  a  game  about  making  hard  resource  allocation  decisions 
in  the  aftermath  of  an  earthquake  might  translate  to  space  travelers  attempting  to  survive  the  rigors  of  an 
alien  planet.  Both  games  could  use  the  same  underlying  game  engine  and  technology,  but  be  presented 
with  different  art  assets  and  supporting  game  mechanics. 

Challenge  2:  Managing  Publicity 

In  the  case  of  Full  Spectrum  Warrior  (released  circa  2004),  a  DoD  program  manager  funded  a 
commercial  company  to  create  a  dual  release  game,  as  a  means  of  creating  a  game  to  train  small  squad 
tactics.  Both  parties  benefited  financially  but  suffered  forms  of  political  backlash  [10].  The  government 
was  criticized  for  funding  a  game  that  was  later  sold  by  a  private  company  for  profit.  The  fact  that  the 
agreement  saved  taxpayer  dollars  did  not  abate  the  criticism.  On  the  other  side,  there  were  reports  of 
controversy  within  the  company  that  developed  the  game,  as  many  managers  felt  that  the  company  should 
not  be  prioritizing  the  creation  of  a  version  of  a  game  that  they  couldn’t  sell  (the  training  version  of  their 
commercial  product).  The  fact  that  they  received  vital  funding  in  advance  to  support  the  additional 
development  costs  did  not  remove  the  controversy.  In  summary,  there  are  social  and  political 
considerations  that  go  along  with  the  potential  benefits  in  cost  and  quality. 

Overall,  this  technique  cannot  be  reliably  depended  upon,  as  there  may  simply  be  no  commercial 
partner  interested  in  making  a  similar  commercial  release.  When  such  opportunities  are  found,  both  sides 
can  potentially  benefit  greatly,  but  even  then  there  is  risk  of  a  backlash.  In  conclusion,  it  is  worth 
watching  for  opportunities  to  leverage  this  model  but  it  should  not  be  relied  on  or  pursued  without  careful 
consideration. 
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5.6  MODEL  F.  SPECIAL  GAME  MODE 


Creating  a  special  game  mode  is  an  approach  that  leverages  existing  commercial  games  for  game- 
based  training  purposes,  by  identifying  such  games  that  already  have  affordance  for  representing 
strategic  choices,  procedural  rehearsal,  or  narrative  immersion  relevant  to  the  objectives  and  training 
skills  that  the  government  agency  is  looking  for. 


Figure  16.  An  example  implementation  of  the  Special  Game  Mode  model. 


The  flow  of  a  special  game  mode  project  is  shown  in  Figure  16.  A  national  laboratory  assesses 
current  training  needs  that  might  be  captured  by  or  embedded  into  existing  commercial  games.  They 
work  with  the  developer  of  that  game  to  design  a  special  game  mode  to  target  those  needs.  The  developer 
takes  over  the  polished  implementation  of  that  mode  and  then  supports  SMEs  and  existing  players  of  the 
commercial  game  to  flesh  out  the  scenario  content  for  that  game  mode.  An  academic  research  team 
validates  that  the  special  game  mode  serves  the  training  need,  and  then  the  training  organization 
integrates  it  into  a  training  curriculum.  The  original  developer  hosts  the  special  mode  using  their 
existing  infrastructure. 

This  model  could  be  similar  to  a  Commercial  Dual  Release  (Model  E);  the  game  mode  created 
might  have  commercial  value  to  the  developer.  However,  even  if  the  special  game  mode  is  only  of  value 
for  training,  reusing  an  existing  and  established  game  can  greatly  reduce  development  costs  (since  many 
iterations  have  already  been  put  into  the  commercial  game),  increase  the  engagement  of  the  product  (since 
the  product  has  already  been  tested  in  a  competitive  market),  and  increase  the  chance  that  the  training 
effectiveness  will  be  validated  (since  we  can  identify  games  that  have  already  demonstrated  the  ability  to 
capture  difficult  strategic  choices,  provide  narrative  immersion,  or  drive  players  to  rehearse  best 
practices). 
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Figure  1 7.  The  roles  each  contributor  might  play  under  a  Special  Game  Mode  type  of  collaboration. 


5.6.1  Examples 

The  Gates  Foundation  funded  an  add-on  to  SimCity  for  learning  [11].  Consider  a  few  other 
potential  special  game  modes  that  could  leverage  popular  games  to  target  relevant  skills  and  decisions: 

•  A  multiplayer  online  battle  arena  (MOBA)  such  as  League  of  Legends,  DOTA  2,  or  Heroes  of 
the  Storm  could  have  a  mode  in  which  human  players  only  play  support  roles,  and  all  other 
champions  are  controlled  through  artificial  intelligence  (AI).  Support  roles  in  those  games 
generally  focus  on  collaboration,  empathy,  and  strategic  awareness,  while  other  roles  focus 
more  on  raw  execution  skill. 

•  A  real-time  strategy  game  (RTS)  such  as  StarCraft  or  Red  Alert  could  be  modified  to  target 
specific  resource  allocation  skills  and  time  management. 

•  A  large-scale  calamity  could  be  injected  into  a  massive  multiplayer  online  (MMO)  game  such 
as  Eve  Online,  allowing  observations  about  how  different  incentives  and  environments  create 
or  undermine  collaboration  by  survivors. 

•  Extensions  to  SimCity  could  be  created  to  focus  on  more  realistic  disaster  preparation,  response, 
and  reconstruction. 

•  A  narrative  game  could  be  created  to  focus  on  the  moral  dilemmas  faced  by  survivors  of  a 
terrorist  attack  as  they  balance  their  own  safety  against  the  opportunity  to  help  others. 

5.6.2  Strengths  and  Weaknesses 

A  special  game  mode  requires  identifying  an  opportunity  for  overlap  between  an  existing  game  and 
a  training  need.  While  this  model  has  many  appealing  features  when  it  is  possible,  sometimes  it  will 
simply  not  be  an  option.  It  is  worth  noting  that  the  types  of  skills  for  which  game -based  training  is 
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particularly  effective  (strategy,  teamwork,  narrative  immersion)  are  skills  that  commonly  appear  in 
successful  entertainment  games.  Consequently,  reusing  an  existing  product  provides  several  clear 
advantages.  Much  of  the  software  already  exists  and  has  been  well  tested,  the  skills  tested  by  the  game  are 
likely  to  be  well  understood  in  fan  analysis  of  the  game,  and  there  is  already  an  existing  player  base  that 
could  be  leveraged  to  help  test  and  refine  new  game  modes.  Fans  of  a  successful  game  are  likely  to  be 
independently  motivated  to  see  their  favorite  game  serve  a  public  good,  and  thus  are  likely  to  effectively 
volunteer  their  time  to  help  refine  new  game  modes.  Furthermore,  some  applications  of  game-based 
training  are  intended  to  train  the  public,  in  which  case  the  existing  player  base  is  especially  important,  as 
they  can  help  overcome  the  major  hurdle  of  bringing  training  efforts  to  the  public’s  attention. 

Perhaps  the  biggest  weakness  of  this  model  is  one  of  acceptance.  In  organizations  that  currently 
have  a  stigma  against  games  used  as  tools  for  training,  reusing  a  game  that  was  never  intended  to  be 
serious  could  be  a  significant  barrier.  Even  if  the  opportunity  is  there,  the  culture  of  the  sponsoring 
agency  may  not  be  willing  to  pursue  it. 

5.7  MODEL  G.  CENTER  OF  EXCELLENCE 

The  proposed  collaboration  models  all  have  strengths  and  weaknesses,  so  picking  the  right  model 
for  a  given  agency  or  a  particular  project  is  a  skilled  task  unto  itself.  Rather  than  leave  these  difficult 
tradeoffs  to  the  sponsoring  agency,  they  could  be  placed  in  the  hands  of  a  new  center  of  excellence  in 
game-based  training.  Such  a  center  might  be  structured  as  an  FFRDC,  an  academic  center  of  excellence, 
or  part  of  a  government  training  agency.  The  center  would  be  a  lasting  repository  of  knowledge  and 
infrastructure  for  making  game-based  training  effective,  thereby  removing  the  need  for  individual 
government  agencies  to  relearn  and  redevelop  those  skills  and  tools  on  each  project. 

Justifying  a  center  of  excellence  would  require  a  certain  critical  mass  of  interest  in  serious  games, 
but  it  may  be  achievable  in  the  near  term  by  leveraging  existing  laboratories  with  serious  games 
experience  and  existing  university  programs  in  game  development  and  modernized  educational 
techniques. 
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6.  SELECTING  THE  BEST  MODEL 


The  seven  models  we  have  discussed  all  have  different  strengths  and  weaknesses.  We  have  already 
eliminated  from  the  list  any  models  that  we  categorically  advise  against;  there  is  no  clear  winner  for  all 
projects  and  all  sponsoring  agencies.  However,  for  a  given  project  in  a  given  agency,  some  models  will  be 
vastly  more  effective  than  others.  For  example,  the  traditional  project  sponsorship  (Model  A)  might  be  the 
best  method  for  creating  additional  scenarios  for  an  existing  training  program,  but  a  game  competition 
(Model  B)  might  be  more  effective  at  creating  materials  to  target  newly  identified  needs  and  gaps. 
Academic  capstone  projects  (Model  D)  might  be  a  good  model  for  agencies  new  to  game-based  training 
but  familiar  with  sponsoring  research,  whereas  a  commercial  dual  release  (Model  E)  or  a  special  game 
mode  (Model  F)  might  be  effective  models  for  agencies  more  comfortable  and  experienced  with  the  use 
of  game-based  training.  A  game  measuring  strategic  skills  might  require  more  careful  oversight  (Model  A 
or  E)  whereas  a  game  about  narrative  immersion  might  benefit  greatly  from  drawing  on  a  wide  range  of 
new  talent  (Model  B  or  D). 

Figure  18  summarizes  the  characteristics  of  each  model,  and  compares  them  to  existing  successful 
models  in  both  government  and  non-government  sectors.  The  rest  of  this  section  provides  further 
information  on  the  model  comparison  relative  to  each  characteristic  (represented  by  the  columns  in  Figure 
18).  The  goal  of  this  section  is  to  provide  the  reader  with  a  starting  point  on  the  types  of  topics  to  consider 
when  choosing  an  appropriate  collaboration  model  for  creating  game-based  training  tools. 
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Project 

Budget 

Initial 

Setup 

Availability  of 
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Impact 

Product 

Maturity 

Similar 

Government 

Structures 

Commercial  Precedent 

A 

Traditional  Project 
Sponsorship 

$500k-10M 

None 

On  Demand 

Low  -  Med 

High 

Sole  Source  Project 

Mission  US,  Mindset 

Quest,  Triad  Interactive 

B 

Game  Design  Competition 

$100k-200k 

None 

On  Demand 

Med  -  High 

Med 

DARPA  Challenge, 

BAA 

Global  Game  Jam,  Game 
Jam  Central, 

lndieGameJams.com 

C 

Project  Bidding  and 
Development  Platform 

$10k-100k 

Required 

On  Demand 

Low  -  Med 

Med 

BAA 

ArtCorgi,  Bricklink, 

Upwork,  Toptal, 

Shapeways 

D 

Academic  Capstone 

$10k-100k 

Required 

On  Demand 

Low  -  Med 

Low 

Existing  Centers  of 
Excellence 

MIT  Beaverworks,  NASA 

Glenn  Research  Center 

E 

Commerical  Dual  Release 

$1M-10M 

None 

Require 

Opportunity 

Med 

High 

Sole  Source  Project 

Full  Spectrum  Warrior, 
America's  Army, 
NumberShire,  Sokikom 

F 

Special  Game  Mode 

$100k-lM 

None 

Require 

Opportunity 

Med 

High 

Sole  Source  Project 

League  of  Legends,  Plants 
vs.  Zombies  2,  Quake, 
Portal  2 

G 

Center  of  Excellence 

$100k-10M 
(varies  by 
sub-model) 

Required 

On  Demand 

High 

Med 

Existing  Centers  of 
Excellence,  FFRDCs 

MassDigi,  CIGG 

Figure  18.  Comparison  of  the  strengths  and  weaknesses  of  the  seven  collaboration  models.  The  final  two  columns 
compare  these  models  to  structures  that  are  tried  and  true  in  both  government  and  non- government  sectors. 
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6.1  APPROXIMATE  PROJECT  BUDGET 


Similar  to  other  software  projects,  the  budget  required  to  create  a  game  varies  greatly  with  the 
intended  scope.  However,  the  collaboration  models  also  play  a  big  factor.  Some  collaboration  models 
only  work  for  large-scale  projects  while  others  only  work  for  less  complex  projects,  so  selecting  the  best 
model  depends  partly  on  the  expected  scale  of  the  program  budget. 

Traditional  project  sponsorship  (Model  A)  involves  a  lot  of  overhead  and  oversight  -  elements  that 
reduce  risk  and  avoid  corruption,  but  also  increase  cost.  Large  programs  can  absorb  such  costs,  but 
smaller  programs  can  be  overwhelmed  by  them,  making  small  lightweight  projects  impossible  in  that 
environment.  Such  projects  generally  select  a  larger  company  with  the  administrative  maturity  to  manage 
government  bureaucracy  and  traceability,  which  also  drives  up  the  cost  of  the  final  project. 

In  contrast,  structures  like  a  game  competition  (Model  B)  can  use  a  relatively  small  amount  of 
funds  to  gain  a  large  range  of  ideas  from  independent  developers  and  fund  only  the  best  design  through 
production;  see  Appendix  B  for  sample  figures  of  how  such  a  program  could  be  structured.  By  working 
with  independent  developers,  the  cost  drops  considerably.  Of  course,  there  are  other  drawbacks,  such  as 
lower  product  maturity,  which  are  captured  in  Figure  18. 

6.2  INITIAL  SETUP 

While  individual  projects  may  have  a  wide  range  of  costs,  some  of  the  suggested  collaboration 
models  require  additional  infrastructure  to  be  provided  up  front  before  any  projects  can  be  pursued  under 
that  model. 

For  example,  a  bidding  platform  (Model  C)  may  have  low  costs  per  project,  but  it  requires  the 
creation  and  maintenance  of  the  platform  for  managing  bids  and  tracking  ratings  for  performers. 
Similarly,  creating  an  individual  capstone  project  (Model  D)  can  be  nearly  free,  apart  from  a  small  level 
of  supervision,  but  such  a  program  requires  a  commitment  on  the  part  of  the  collaborating  university  that 
likely  comes  with  up-front  costs  for  helping  them  to  design  their  program  to  align  with  government  needs. 

Using  an  existing  center  of  excellence  (Model  G)  for  managing  game-based  training  might 
minimize  setup  costs  (in  comparison,  for  example,  to  creating  an  entirely  new  center),  but  even  an 
existing  center  can  have  costs  outside  of  the  individual  projects  -  for  a  center  of  excellence  to  be 
effective,  it  needs  to  have  an  active  repository  of  experience  and  talent,  which  may  involve  gap  funding  to 
carry  the  center  along  between  programs,  or  provisions  to  allow  them  alternative  sources  of  funding 
beyond  government  programs. 

In  contrast,  the  other  models  can  be  run  on  a  per-project  basis.  While  the  cost  may  be  high  for  a 
traditional  project  model  (Model  A)  or  a  dual  release  commercial  project  (Model  E),  that  cost  is  limited  to 
the  individual  programs.  No  outside  infrastructure  (neither  organizational  nor  technological)  is  required  to 
enable  such  programs. 

6.3  AVAILABILITY  OF  OPPORTUNITIES 

Some  models  can  be  executed  whenever  there  is  a  demand  for  them.  Others  require  finding  a 
suitable  opportunity,  but  the  availability  of  such  opportunities  cannot  be  fully  predicted.  This  limitation 
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applies  most  strongly  to  the  models  that  rely  on  leveraging  commercial  products  that  are  similar  to  the 
domain  or  skills  being  targeted,  either  the  creation  of  a  new  product  (via  a  dual  release)  or  modification  of 
an  existing  one  (via  a  special  game  mode).  This  limitation  is  not  to  say  that  those  opportunities  are  rare; 
they  may,  in  fact,  be  common.  However,  regardless  of  their  frequency,  they  represent  an  element  of 
uncertainty  that  can  limit  the  use  of  the  model. 

The  other  models  are  largely  free  of  such  limitations.  Even  the  bidding  platform  (Model  C)  can 
avoid  any  such  issue.  The  software  freelance  and  independent  developer  communities  have  a  large  supply 
of  skilled  workers  who  are  willing  and  able  to  meet  the  requests  made  on  a  bidding  platform.  Even  if  the 
platform  is  only  lightly  used  and  thus  is  only  providing  nominal  income  to  the  developers  serving  it,  it  is 
likely  to  have  a  responsive  community.  Similar  platforms  for  other  low-salary  industries  (such  as  artists) 
have  a  track  record  of  providing  responsive  service,  even  when  they  are  only  serving  as  supplemental 
income  for  the  participants.  They  may  be  unreliable  for  suppliers  of  the  products,  but  they  are  typically 
very  reliable  for  customers.  In  the  case  of  capturing  independent  game  developers,  the  government  would 
benefit  as  a  customer  receiving  reliable  service. 

6.4  LIKELIHOOD  OF  IMPACT 

Many  factors  aside  from  the  collaboration  model  contribute  to  whether  or  not  a  program  ultimately 
has  a  positive  impact,  and  all  programs  carry  risk  of  falling  short.  However,  some  models  mitigate  that 
risk  more  than  others.  This  risk  is  not  referring  to  the  risk  of  fraud  or  deception  on  the  part  of  the 
performer,  but  rather  the  likelihood  that  the  product  will  actually  solve  the  customer’s  problem.  It  is  all 
too  common  for  a  developer  to  provide  the  exact  product  requested,  only  for  the  buyer  to  later  discover 
that  their  request  was  not  quite  what  they  wanted. 

There  are  two  big  factors  in  the  likelihood  that  a  product  will  fail  to  solve  the  customer’s  problem 
despite  meeting  its  legal  specifications: 

•  the  number  of  design  alternatives  assessed,  and 

•  the  creative  freedom  and  motivation  of  the  performer. 

These  factors  are  even  more  important  in  user-facing  software  such  as  games  and  game-based  training 
materials. 

A  traditional  project  sponsorship  (Model  A)  typically  suffers  on  both  accounts.  It  quickly  narrows 
the  design  space  down  to  a  single  option,  without  iterating  on  alternative  designs,  and  it  provides  a  legally 
binding  requirements  document  that  limits  the  freedom  of  the  performer  to  modify  the  design  as 
development  proceeds.  Those  are  often  necessary  costs  to  minimize  opportunities  for  corruption  and 
fraud,  but  they  are  still  very  real  costs. 

A  project  bidding  platform  (Model  C)  suffers  similar  problems.  In  order  for  developers  to  bid  on  a 
project  request,  they  need  to  know  the  detailed  design  of  that  project  and  the  expectations  of  the  buyer. 
However,  by  providing  that  information  up  front  and  then  selecting  a  bidder,  the  project  has  already 
added  a  risk  that  the  project  will  fail  to  meet  their  true  needs. 
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A  game  design  competition  (Model  B)  rates  fairly  well.  The  independent  development  community 
has  a  reputation  for  self-motivated  talent  and  any  underperformers  are  easily  rooted  out  in  a  competition 
with  enough  participants.  A  relatively  small  prize  pool  can  motivate  a  large  number  of  participants  to 
contribute  ideas  and  prototypes,  allow  the  customer  a  wide  range  of  options,  allow  those  options  to  be 
partially  developed,  and  be  very  selective  about  which  ideas  are  pursued  to  completion.  Between  a  high 
number  of  design  alternatives  and  developers  with  creative  freedom,  there  is  an  increased  chance  that  the 
final  design  will  be  effective  and  impactful. 

Academic  capstone  projects  (Model  D)  do  not  rate  as  well,  despite  superficial  similarities  to  a  game 
design  competition.  While  students  are  likely  to  have  many  ideas,  they  lack  the  experience  and  wisdom  to 
vet  those  ideas  themselves.  So,  while  there  will  be  a  larger  number  of  ideas  produced,  there  will  likely  be 
a  lower  number  of  good  ones.  While  capstone  projects  are  inexpensive  and  may  produce  good  ideas,  they 
will  never  be  as  reliable  as  professionals,  even  independent  professionals. 

Keep  in  mind  that  while  these  models  differ  in  their  chance  of  success,  all  of  them  are  reasonable  to 
consider.  Models  that  would  usually  produce  an  unusable  product  have  already  been  omitted  from  the  list. 
The  ratings  are  relative,  and  benefits  in  other  columns  (such  as  cost,  maturity,  or  opportunity)  may  make 
up  for  a  deficiency  in  this  one. 

6.5  PRODUCT  MATURITY 

We  distinguish  the  likelihood  of  impact  from  the  maturity,  stability,  and  professionalism  of  the 
product  itself.  The  quality  of  the  product,  when  it  does  succeed,  is  largely  a  function  of  the  experience  of 
the  performer.  Amateurs  (such  as  students)  will  produce  rough  (albeit  usually  functional)  products,  small- 
scale  professionals  will  produce  solid  products  (such  as  independent  developers  in  a  game  competition  or 
freelance  software  developers  using  a  bidding  platform),  and  the  best  products  will  come  from  developers 
with  a  proven  track  record. 

6.6  SIMILAR  GOVERNMENT  STRUCTURES 

Some  of  the  models  may  appear  novel  or  unusual  for  a  government-sponsored  project,  but  they  all 
have  analogs  with  long  and  reputable  histories.  Traditional  sponsorship,  commercial  dual  releases,  and 
special  game  modes  are  really  just  variations  of  sole-source  projects.  They  offer  different  means  of 
reducing  the  cost,  encouraging  creative  freedom,  or  leveraging  existing  user  bases  and  infrastructure,  but 
they  do  not  fundamentally  differ  from  most  government-sponsored  projects. 

Game  design  competitions  and  bidding  platforms  have  a  history  in  DARPA  challenges  and  are 
essentially  variations  of  traditional  BAA  models.  They  are  tailored  to  capture  talent  from  the  games 
industry,  but  would  unfold  in  a  manner  familiar  to  many  government  agencies. 

Academic  capstone  projects  and  creating  new  centers  of  excellence  are  similar  to  existing  models 
where  government  agencies  fund  academic  centers  of  excellence,  FFRDCs,  and  research  conferences  to 
create  communities,  improve  research,  and  direct  industry  toward  problems  relevant  to  that  agency. 
Those  approaches  are  certainly  well  suited  for  agencies  with  a  research  focus,  but  they  can  also  be  very 
effective  tools  for  risk  reduction  and  coming  up  with  fresh  ideas  for  projects  that  aim  simply  to  produce  a 
useful  product. 
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APPENDIX  A.  SERIOUS  GAMES  FOCUS  GROUP  NOTES 


On  August  27,  2015,  MIT  Lincoln  Laboratory  hosted  a  focus  group  about  serious  games  at 
Beaver  Works  on  MIT  Campus  in  Cambridge,  MA.  The  group  focused  on  overcoming  practical  and 
organizational  barriers  to  collaboration  between  government  organizations  with  existing  or  emerging 
interest  in  serious  games  and  nongovernment  organizations  that  might  be  able  to  support  those  efforts. 
There  were  over  25  attendees  from  a  range  of  organizations  interested  in  serious  games,  representing 
multiple  organizations  from  each  of  the  following  economic  sectors: 

•  Government  agencies,  including  both  project  managers  and  Incident  Management  Assistance 
Teams  (IMATs), 

•  National  laboratories,  including  multiple  FFRDCs, 

•  Academic  institutions,  including  both  game  development  degree  programs  and  cognitive 
science  researchers,  and 

•  Video  game-related  private  companies,  including  large  companies,  independent  developers, 
and  nonprofit  research-based  firms. 

The  working  group  was  organized  around  a  mixture  of  full  group  discussions  and  breakout 
sessions;  each  breakout  group  contained  four  to  six  attendees  spanning  all  four  economic  sectors. 

The  information  captured  in  this  appendix  supports  and  elaborates  on  the  ideas  of  the  main 
document.  It  serves  both  as  corroborating  evidence,  and  as  suggestions  of  future  directions  for  extending 
these  ideas.  Each  paragraph  in  the  rest  of  this  appendix  is  attributed  to  an  economic  sector;  indicating  the 
background  of  the  attendees  who  contributed  the  idea.  Additional  citations  and  explanation  have  been 
added  by  the  authors  of  this  document  to  substantiate  statements  made  by  the  attendees,  but  the  ideas 
described  in  this  appendix  were  generated  by  attendees  not  the  authors.  These  comments  have  been 
reviewed  with  those  attendees  to  ensure  that  they  are  accurate  and  authentic. 

OPPORTUNITIES  FOR  IMPACT 

Game-based  training  can  be  applied  to  a  range  of  topics.  Helping  experienced  tactical  thinkers 
understand  strategic  tradeoffs  and  experience  larger-scale  dilemmas  prior  to  an  actual  incident  is  valuable 
for  supporting  emergency  managers  as  they  rise  in  rank.  Their  on-the-job  experience  and  traditional 
training  methods  serve  them  well  in  developing  their  tactical  and  procedural  skills,  but  they  often  struggle 
when  they  advance  to  a  position  requiring  a  larger  view  of  the  situation.  Strategic  game-based  techniques 
can  bring  those  issues  to  the  surface  and  help  to  directly  address  them  [12].  <Source:  Government 

Game-based  methods  can  also  be  an  effective  and  lower-cost  method  for  targeting  skills  currently 
taught  in  the  classroom  or  via  a  live  exercise  [13].  For  example,  one  concern  with  live  training  and 
rehearsal  is  that  human  behavior  can  change  in  the  presence  of  an  observer,  distorting  assessments  of 
readiness  and  preparation.  In  contrast,  background  monitoring  of  a  digital  exercise  over  multiple 
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iterations  of  a  short  scenario-based  game  can  put  people  at  ease,  get  a  more  accurate  measurement,  and 
provide  more  repetitions  to  improve  retention.  <Source:  Government  > 

A  valuable  part  of  any  educational  tool  is  feedback.  It  is  well  accepted  that  shorter  feedback  cycles 
engage  and  train  students  more  effectively  in  both  entertainment  and  educational  settings.  The  briefing 
after  a  live  exercise  can  be  just  as  valuable  as  the  scenario  itself,  as  it  offers  a  chance  to  reflect  and  discuss 
decisions,  mistakes,  and  triumphs  just  experienced.  Game -based  training  offers  an  opportunity  to  make 
that  feedback  loop  shorter  (giving  feedback  as  you  play  the  game)  and  allow  more  iterations  of  the 
learning  cycle  (by  playing  a  shorter  training  game  several  times  in  sequence).  <Source:  Academia> 

Playing  nominally  one -player  games  as  a  group  can  drive  a  discussion  that  is  informative  and 
supportive  for  creating  cohesive  teams.  Being  forced  to  justify  and  explain  a  move,  rather  than  just 
making  it,  can  recreate  valuable  dynamics  from  real  situations  in  the  context  of  even  very  simple  game- 
based  training  scenarios.  <Source:  Government 

Another  part  of  training  is  emotionally  understanding  the  people  involved  in  an  incident,  including 
survivors,  other  agencies,  and  political  figures.  Games  and  interactive  fiction  have  matured  greatly  over 
the  past  two  decades  in  delivering  narrative  experiences  that  immerse  the  player  and  trigger  empathy  and 
emotion  toward  entirely  fictional  characters  by  drawing  on  techniques  from  film  combined  with  involving 
the  players  themselves  in  the  story.  Narrative  games  could  be  leveraged  to  capture  formative  experiences 
of  experienced  professionals,  and  to  crystalize  and  convey  these  to  the  next  generation  of  responders  in  a 
way  that  sinks  in  and  has  a  lasting  impression.  One  attendee  said,  “For  emergency  response  training,  you 
want  to  create  emotional  attachment,  so  players  feel  the  pain  of  hard  choices  and  loss,  but  to  still  have  a 
fictional  environment  they  can  later  separate  themselves  to  reflect  and  think  strategically.”  Games  have 
proven  their  ability  to  do  that,  and  narrative-style  games  should  be  added  to  the  suite  of  tools  used  by 
professional  trainers.  <Source:  Academia> 

Another  class  of  tools  that  could  be  of  value  to  government  agencies  are  public-facing  games,  with 
the  goal  of  assessing  public  preparedness  and  predicting  public  response  to  scenarios  of  interest.  For 
example,  “survival”  games  are  quite  popular  right  now,  involving  unarmed  players  attempting  to  survive 
in  hostile,  and  often  quite  realistic,  environments.  Data  from  such  a  game  could  reveal  how  people 
respond  to  novel  situations,  form  collaborations,  and  react  to  public  communication.  A  sequel  to  such  a 
game,  released  just  after  a  real  world  event,  could  also  measure  how  the  public  responds  differently  when 
they  are  primed;  e.g.,  add  a  forest  fire  module  to  a  survival  game  after  new  coverage  of  forest  fires  or  an 
educational  campaign  by  FEMA.  Even  if  the  game  is  only  played  by  a  small  sliver  of  the  nation’s 
population,  it  could  provide  a  valuable  insight  into  what  efforts  and  events  change  people’s  behaviors. 
You  could  also  correlate  such  behavior  to  other  measures  of  the  players’  personality  archetypes,  to  better 
understand  how  to  effectively  provide  guidance  and  education  to  different  segments  of  the  public. 
<Source:  Industry> 

GARNERING  GOVERNMENT  INTEREST 

Despite  a  range  of  potential  benefits  (lightweight  training,  soft  skill  evaluation,  expert  knowledge 
transfer  via  interaction  scenarios,  strategic  exploration),  many  parts  of  DHS  and  other  government 
agencies  hesitate  to  seriously  consider  serious  games  as  tools  to  leverage.  There  is  still  a  stigma  against 
games  as  being  “frivolous”  or  “unprofessional.”  Such  perceptions  persist  despite  a  long  history  of  games 
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in  DoD  [14,  15,  16,  17]  and  active  research  showing  their  potential  benefits  [1,  3,  4].  While  there  are  open 
research  questions  about  the  use  of  serious  games,  and  when  and  how  they  can  be  effective,  many  of  the 
obstacles  are  political  and  social  not  scientific  or  technical.  However,  there  are  a  number  of  strategies  to 
overcome  such  barriers.  <Source:  Government 

Continuing  research  on  providing  quantitative  measurements  of  benefits  can  help  remove  political 
and  social  barriers.  While  traditional  training  techniques,  such  as  classroom  or  online  training,  are  often 
implemented  based  on  best  judgment  of  professionals,  the  newness  of  serious  games  puts  them  under 
greater  scrutiny  and  requires  stronger  scientific  validation.  Measuring  educational  outcomes,  especially  in 
the  area  of  soft  skills  and  strategic  skills,  has  always  been  difficult,  but  other  types  of  validations  can  help 
build  confidence.  For  example,  game -based  training  validation  could  benefit  from  more  work  in  the  areas 
of  knowledge  retention  studies,  informal  self-reports  from  students  and  instructors,  tracking  time 
voluntarily  spent  with  the  training  materials,  and  measuring  types  of  conversations  generated  within 
teams.  <Source:  National  Laboratory> 

Terminology  matters  a  lot.  A  method  called  a  “serious  game”  might  appear  unprofessional,  whereas 
the  same  method  labeled  as  “virtual  training,”  “modernized  training,”  or  “digital  exercises”  might  be 
accepted.  Similarly,  analogies  between  training  methods  and  video  games  might  be  intimidating  to  users 
who  are  not  computer  savvy,  whereas  the  same  users  likely  have  smartphones  and  would  be  quite 
comfortable  with  the  same  material  labeled  as  an  “app.”  Overcoming  first-impression  biases  can  be  an 
important  part  of  gaining  traction.  <Source:  Government 

It  is  often  difficult  to  convince  people  that  a  Chemical  Biological  Radiological  Nuclear  Explosive 
(CBRNE)  event  is  an  important/hard  problem  when  an  incident  has  never  occurred  or  when  the  incidents 
are  very  rare.  Lower-cost  techniques  for  preparation,  such  as  game -based  training,  could  interest  local 
branches  with  many  pressures  on  their  time.  National  FEMA  operates  to  support  the  regions;  if  the 
regional  coordinator  can  demonstrate  need  (including  data  to  support  skill  transfer  effectiveness), 
National  FEMA  can  provide  resources  to  do  so  and  will  be  much  more  amenable  to  newer  techniques 
such  as  game-based  training.  Collaborations  with  local  and  regional  agencies  also  provide  opportunities  to 
continue  the  kind  of  validation  research  that  is  important  to  mature  the  field.  Local  and  regional  agencies 
could  track  student  feedback  and  impact  on  retention  after  integrating  game-based  learning  into  classroom 
and  online  training  courses.  <Source:  Government 

In  fact,  regardless  of  the  method  employed  (game-based  or  otherwise),  training  organizations  will 
need  to  improve  their  understanding  of  the  targeted  soft  skills.  For  example,  game-based  and  other 
modernized  training  methods  will  rely  on  clarifying  the  following  elements: 

•  How  do  we  identify  a  set  of  skills  relevant  to  a  particular  community  or  position? 

•  How  do  we  evaluate  or  test  the  presence  of  a  skill? 

•  How  do  we  train  an  emergency  manager  to  move  from  one  position  to  the  next  (e.g.,  from 
regional  to  national)? 

•  How  do  we  ensure  that  skills  transfer  from  the  training  to  job  competence? 
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Without  well-understood  and  objective  measurements  of  job  performance,  it  is  difficult  to  properly 
validate  or  assess  any  training  method,  game-based  or  otherwise.  It  is  unrealistic  to  expect  such  standards 
to  emerge  in  the  near  future,  so  alternative  approaches  should  be  investigated,  such  as  informal  SME 
validation  of  game -based  training  materials  rather  than  formal  scientific  validation.  <Source: 
Government 

Many  government  sponsors  are  less  worried  about  carefully  validating  skill  transfer  and  more 
worried  about  whether  the  users  (i.e.,  trainers,  management,  students)  will  accept  the  game,  and  if  it  will 
be  built  in  a  way  that  draws  on  relevant  domain  expertise  and  is  cognizant  of  the  target  students’  needs. 
<Source:  National  Laboratory,  Government 

Initially,  it  might  be  prudent  to  focus  on  games  that  assess  a  skill  but  do  not  necessarily  strive  to 
teach  it.  Assessment  games  could  be  a  good  first  step  in  getting  more  government  agencies  to  be 
comfortable  with  the  use  of  serious  games.  It  would  be  a  smaller  role  in  a  training  program,  while 
simultaneously  providing  a  platform  for  collecting  data  to  validate  the  effectiveness  of  games  at  targeting 
training  goals  and  relevant  skills.  <Source:  Government 

COMMERCIAL  COLLABORATION  OPPORTUNITIES 

Some  of  the  collaboration  models  proposed  would  involve  funding  an  organization  to  build  a  game 
with  a  large  budget.  For  those  models,  the  funding  is  the  primary  motivation  for  participation.  However, 
other  models  rely  more  on  alternative  motivation  for  participation  that  can  greatly  reduce  the  financial 
cost  of  the  project.  For  example,  if  participants  in  a  competition  or  capstone  project  can  keep  their 
intellectual  property  (IP)  for  commercial  use,  or  if  they  can  reuse  some  of  the  infrastructure  they  develop 
for  subsequent  projects,  then  a  small  budget  for  a  government  training  project  can  become  very  appealing. 
A  $10k-$50k  budget  that  would  look  small  as  a  means  of  sustaining  a  company  in  the  long  run  can 
become  a  valuable  seed  investment  if  some  of  the  output  of  that  project  can  be  reused  for  private 
purposes.  For  example,  a  developer  might  build  a  training  tool  that  involves  3D  first-person  interaction 
with  a  disaster  zone,  then  later  reuse  the  3D  engine,  scoring  mechanics,  character  AI,  and  networking 
code  for  a  survival  game,  a  first-person  shooter,  or  a  fantasy-themed  adventure.  Even  if  the  game  itself  is 
focused  on  serious  purposes,  much  of  the  software  infrastructure  and  art  assets  could  be  reused,  offering  a 
nonmonetary  incentive  for  participation.  <Source:  Industry> 

Sometimes,  just  adding  a  thin  layer  on  top  of  an  educational  simulation  can  turn  a  boring 
experience  into  an  engaging  strategy  game.  Keep  in  mind  that  many  entertainment  games  that  are  played 
purely  for  “fun”  can  involve  managing  massive  operations  research  supply  chains,  balancing  multi-object 
optimization  pressures,  social  interaction  and  political  maneuvering  with  other  players,  and  a  host  of  other 
activities  that,  to  an  outsider,  sound  more  like  work.  One  of  the  lessons  from  the  success  of  strategy 
games  is  that  hard  decisions  embedded  in  a  believable  world  engage  and  motivate  players,  and  such 
games  typically  feature  decisions  and  tradeoffs  that  are  near-perfect  matches  to  the  decisions  and  skills 
described  by  professional  emergency  managers  as  being  core  to  a  successful  response.  In  fact,  the 
category  of  “strategy  games”  is  often  defined  as  a  game  where  players  are  given  impactful  decisions  that 
do  not  have  known  correct  answers,  and  for  which  the  best  answer  depends  heavily  on  context  [18]. 
<Source:  Industry,  Academia  > 
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In  some  cases,  temporary  access  to  professional-grade  software  development  tools  is  an  incentive 
for  participation.  For  example,  many  commercial  game  competitions  arrange  a  deal  with  the  Unity  tool 
providers  (or  similar  game  engine)  to  give  all  participants  a  temporary  license  for  the  professional  version 
of  the  software.  Many  small  developers  see  participation  as  an  opportunity  to  learn  those  tools  and  build 
their  skills,  as  well  as  a  chance  to  earn  prize  money  and  add  to  their  portfolio.  <Source:  Industry> 

There  is  a  large  pool  of  inexpensive  talent  available  in  the  independent  video  game  community, 
which  many  of  the  models  described  in  this  document  aim  to  leverage.  Similarly,  there  is  inexpensive 
talent  that  can  be  accessed  for  art  and  sound,  both  of  which  are  easily  overlooked  but  critical  elements  of 
creating  an  engaging,  believable,  and  immersive  experience  for  the  player  or  student.  For  example, 
students  at  the  Berkeley  School  of  Music  are  world-class  musicians  who  are  often  available  to  help  with 
game-related  projects  at  very  low  cost.  <Source:  Industry> 

Consider  involving  the  Higher  Education  Video  Game  Alliance  (HEVGA)  [19].  They  have 
addressed  similar  issues  of  how  to  leverage  games  for  advanced  education,  and  could  serve  as  a  kind  of 
center  of  excellence  for  supporting  similar  efforts  with  government  funded  training  programs.  <Source: 
Academia> 

It  will  likely  be  very  hard  to  get  even  small  amounts  of  engineering  time  from  medium-to-large 
video  game  companies  without  committing  to  high  funding  up  front.  They  might  be  willing  to  share  their 
existing  data,  but  unlikely  to  add  a  new  data  collection  field  or  collect  a  new  kind  of  data.  It  might  be 
possible  to  have  industry  share  data  with  a  national  laboratory,  who  can  collate  it  into  a  common  form  in  a 
trusted  environment,  and  then  share  anonymized  aggregate  results  with  academia  for  further  study.  In 
contrast,  small  and  independent  developers  are  likely  to  be  very  interested  and  available,  making  them  an 
important  resource  to  learn  to  leverage  on  the  government  side.  <Source:  Industry> 

There  can  also  be  fears,  founded  or  unfounded,  of  risks  that  taking  government  money  might  entail, 
especially  if  trying  to  create  a  commercial  dual  release  (Model  E).  They  might  be  forced  to  add  or  remove 
features  by  their  sponsor  that  leave  the  game  too  boring  for  commercial  success,  or  their  resources  might 
be  entirely  occupied  in  meeting  the  government  contract,  leaving  them  little  time  to  properly  design  the 
commercial  version,  but  legally  liable  if  they  fall  short  on  the  government  version.  For  example,  the 
company  38  Studios  was  sued  for  over  $4  million  after  taking  government  funding  and  then  struggling  to 
release  their  product.  Game  projects  always  carry  risk  with  them,  and  the  cost  of  failure  is  usually  wasting 
the  time  and  effort  that  went  into  their  creation.  Adding  legal  penalties  for  contract  violation  on  top  of  that 
risk  could  deter  otherwise  qualified  participants.  A  model  more  like  a  game  competition  where 
unsuccessful  products  are  weeded  out,  but  not  legally  penalized,  would  help  alleviate  such  concerns. 
<Source:  Industry> 

LESSONS  FROM  COMMERCIAL  GAMES 

Commercial  game  companies  offer  more  than  just  experience  in  building  the  software  behind 
games,  often  at  a  low  cost.  They  also  offer  motivated  and  experienced  professionals  who  have  thought 
hard  about  engaging  users  while  teaching  them  new  systems.  Modern  video  games  are  often  complex 
systems  that  take  many  hours  to  learn,  and  one  of  the  triumphs  of  the  industry  is  in  learning  how  to  teach 
nonexperts  (new  players)  how  to  use  complex  systems  (the  game),  make  long-range  strategic  choices 
about  those  systems  (playing  the  game),  and  display  large  amounts  of  data  in  an  accessible  fashion  (the 
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interface).  They  manage  to  routinely  succeed  on  those  fronts  with  an  audience  that  is  not  only  free  to  quit 
at  any  time,  but  which  is  actually  paying  to  participate.  Those  lessons  have  obvious  application  to  the 
field  of  emergency  management  training.  <Source:  Industry> 

The  video  game  industry  has  amassed  techniques  and  skill  at  making  games  that  are  engaging 
(critical  to  education  as  well  as  entertainment)  and  effective  at  building  communities.  It  is  common  for 
player-driven  discussion  forums  to  spring  up  online  based  on  commercial  video  games,  where  players 
help  each  other  improve  their  gameplay  and  master  the  game.  Similarly,  “modding”  communities  are 
common  for  successful  games,  where  players  create  new  content  and  elaborate  scenarios  for  games  they 
enjoy.  In  both  cases,  they  volunteer  their  time  to  improve  a  game  they  paid  for,  simply  because  they  find 
it  engaging.  In  the  realm  of  training  emergency  professionals,  there  is  already  a  highly  motivated  user 
base  who  cares  deeply  about  the  topic.  Game-based  training  methods  could  provide  them  an  opportunity 
and  a  spur  to  document  their  experiences,  discuss  them,  and  collaborate  across  the  country  in  improving 
their  performance  at  those  games.  As  that  user  base  is  filled  more  and  more  with  people  who  grew  up 
with  video  games,  game-based  training  can  provide  a  natural  and  familiar  way  for  them  to  share 
knowledge  and  capture  scenarios  from  their  experience.  <Source:  Industry> 

There  have  already  been  cases  of  knowledge  sharing  occurring  in  educational  spaces.  The  EDGE 
game  created  by  DHS  and  U.S.  Army  addressed  active  shooter  scenarios  in  an  online  fashion  very  similar 
to  an  MMO.  Users  of  that  program  ended  up  building  an  online  community  where  players  shared 
techniques,  discussed  real-world  experiences,  and  forged  interdistrict  and  interstate  social  connections 
and  collaborations  [20,  21].  One  of  the  benefits  of  live  exercises  is  the  forging  of  social  and 
organizational  connections,  and  game-based  training  offers  another  tool  toward  that  end.  <Source: 
Government 

One  lesson  from  commercial  video  games  is  in  how  one  player  games  and  multiplayer  games  offer 
different  incentives  that  can  drive  different  behavior  and  motivate  different  types  of  players.  When  a 
human  faces  another  human  in  a  game,  they  are  often  trying  to  one-up  the  other  player  -  which  may 
involve  showing  off  skills  more  than  actually  focus  on  winning.  When  facing  a  machine  AI,  players  are 
just  trying  to  win,  and  often  focus  much  more  on  learning  good  strategies.  Choosing  one  or  the  other  type 
of  game  can  help  motivate  students  using  a  game-based  training  tool,  or  encourage  them  to  focus  on 
elements  that  are  important  to  the  lessons.  For  example,  a  competitive  player-versus-player  mode  can  be 
effective  for  communities  (such  as  air  traffic  control)  with  competitive  employees,  while  solo  games 
might  be  more  suited  to  communities  (such  as  public  health)  that  typically  employ  people  less  interested 
in  one-upping  their  colleagues.  Characterizing  the  kind  of  student  (and  thus  the  kind  of  player)  can  draw 
on  lessons  learned  from  the  commercial  world  and  apply  them  to  professional  training  [22].  <Source: 
Industry> 

There  is  a  related  lesson  about  tailoring  a  game’s  look  and  feel  to  the  target  audience’s  expectations 
and  familiar  environment.  Some  people  are  going  to  prefer  a  display  with  a  high  density  of  numbers  and 
controls,  feeling  like  they  are  managing  a  complex  system  with  detailed  realism.  Others  may  want  a 
cleaner  interface  that  uses  natural  actions  (e.g.,  walking  around  in  a  3D  environment  instead  of  selecting  a 
destination  from  a  menu).  <Source:  Academia> 
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TOOL  SUPPORT  FOR  GAME-BASED  TRAINING 


Part  of  making  game -based  training  techniques  inexpensive,  regardless  of  the  collaboration  model, 
is  providing  a  toolkit  or  library  of  reusable  algorithms.  For  example,  Oak  Ridge  National  Laboratory 
already  has  some  models  of  how  populations  are  likely  to  behave  in  disaster  scenarios.  Both  Oak  Ridge 
and  Lawrence  Livermore  laboratories  have  models  of  the  lethality  of  various  hazards,  such  as  the 
radiation  protection  factors  of  various  building  types.  Lincoln  Laboratory  also  has  models  and  game 
templates  that  can  be  reused  in  a  range  of  domains.  Allowing  the  use  of  such  tools  outside  of 
government  development  would  require  additional  authorization,  but  there  is  a  common  practice  and 
procedure  for  open  sourcing  materials  for  use  in  a  public  community  when  it  is  deemed  to  support 
national  interests.  <Source:  National  Laboratory> 

Crowdsourcing  talent  from  a  pool  of  independent  developers  or  volunteers  from  the  player 
community  is  quite  possible,  but  note  that  they  will  need  some  enablers.  Two  big  enablers  that  will  need 
to  be  provided  as  part  of  the  project  are  (a)  domain  knowledge  and/or  tools  to  provide  realistic  data  about 
elements  of  the  domain,  and  (b)  a  platform  or  framework  for  hosting  and  sharing  the  game.  Many 
training  organizations  have  both  on  hand,  so  the  main  challenge  is  how  to  fold  them  into  the  project 
structure  so  that  they  are  involved  early.  <Source:  National  Laboratory> 

A  common  concern  with  game-based  training  is  whether  or  not  the  games  are  realistic,  affecting 
whether  or  not  their  lessons  will  transfer  to  real-world  operations.  Games  developed  with  an  iterative 
process  offer  an  easy  solution  to  this:  having  them  played  by  domain  experts  early  in  the  development 
cycle,  so  that  the  designers  and  developers  get  feedback  on  the  game.  Involving  subject  matter  experts  in 
the  creation  of  educational  materials,  rather  than  relying  on  scientific  studies,  is  a  common  practice  for 
creating  classroom  and  online  training  materials,  and  the  same  techniques  could  be  applied  to  game- 
based  training  [23].  It  is  rare  to  hear  a  classroom  course  criticized  for  not  being  validated  by  a  double 
blind  study  of  its  impact,  but  that  is  a  common  concern  raised  with  serious  games.  While  training 
materials  should  certainly  be  validated,  that  might  be  best  achieved  by  tightly  involving  SMEs  rather 
than  attempting  scientific-style  validation  after  the  game  has  already  been  created  (and  paid  for).  <Source: 
Academia,  Government 

THE  NEED  FOR  ALTERNATIVE  COLLABORATION  MODELS 

It  is  worth  noting  that  effective  games,  be  they  for  entertainment  or  training,  require  engaging  the 
player.  Simply  adding  a  score  to  a  simulation,  or  adding  “badges”  to  an  exercise  or  quiz  does  not 
constitute  an  effective  game -based  training  method.  The  game  needs  to  integrate  the  material  into  a 
decision  space,  with  the  rewards  being  part  of  the  game  and  not  superficially  added  as  an  afterthought. 
Garnering  that  engagement  is  critical,  and  is  an  area  where  the  video  game  industry  has  a  long  and 
successful  history.  Collaborating  with  video  game  developers,  not  just  generic  software  developers,  will 
be  critical  to  creating  effective  game -based  training.  <Source:  Industry> 

The  traditional  sponsorship  model  is  far  too  linear.  It  is  not  how  games,  or  really  any  type  of 
software,  is  built  in  industry.  While  the  other  models  may  appear  unfamiliar  to  government  agencies,  they 
should  be  taken  seriously.  Also,  many  of  the  models  do  have  successful  precedents  in  government 
projects,  even  if  they  are  not  the  dominant  style  of  project  [6].  <Source:  Industry,  Government 
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A  common  stereotype  that  carries  far  too  much  truth  in  it  is  that  anything  made  by  the  government 
will  be  boring,  hard  to  use,  and  overpriced.  To  captivate  an  audience  -  key  to  making  educational 
materials  stick  -  it  will  be  necessary  to  leverage  private  companies  that  have  lived  or  died  on  their 
abilities  to  keep  players  engaged.  The  video  game  industry  is  full  of  such  companies,  and  they  can  be  a 
valuable  asset  to  government  programs  producing  user-facing  tools,  such  as  training  materials.  The 
successful  companies  understand  how  to  build  engagement  into  a  product  up  front,  rather  than  try  to  add 
it  later  or  rely  on  a  captive  audience.  <Source:  Industry> 

Rather  than  attempting  to  build  a  full  system,  training  organizations  should  consider  creating  a  core 
engine  by  conventional  means  (e.g.,  traditional  project  sponsorship,  Model  A)  and  then  using  some  of  the 
other  proposed  collaboration  models  to  specialize  that  core  system  to  the  needs  of  individual  training 
organizations.  This  approach  would  lower  the  barrier  of  entry  for  independent  and  small-scale 
collaborators  from  industry,  giving  them  a  way  to  contribute  their  talents  without  getting  bogged  down  in 
a  software  project  too  big  for  their  team.  A  training  organization  could  provide  the  core  infrastructure, 
such  networking,  graphics,  and  game  engines  up  front,  and  then  have  the  games  industry  collaborators 
build  scenarios  and  extensions  to  the  core  system  focused  on  individual  training  organizations’  needs.  It 
would  greatly  lower  the  cost  of  individual  projects,  and  allow  a  much  wider  range  of  contributing 
companies  to  compete  for  those  projects.  The  end  result  would  be  lower  cost  and  higher  quality,  with  a 
modest  up-front  investment.  <Source:  Industry> 

Expecting  individual  agencies,  especially  relatively  small  training  organizations,  to  build  game- 
based  training  from  scratch  on  each  project  may  be  unrealistic.  Building  effective  games  is  a  skill  built  up 
over  time,  and  it  would  be  inefficient  to  have  every  game-based  training  project  rediscover  that  skill  and 
reencounter  the  pitfalls  associated  with  the  field.  Having  a  persistent  organizing  force  (e.g.,  a  center  of 
excellence,  Model  G)  could  be  extremely  valuable  to  address  a  range  of  potential  issues: 

•  Carry  over  experience  between  projects,  so  that  lessons  learned  are  not  re-learned  each  time, 

•  Provide  domain  knowledge  to  guide  collaborating  organizations  unfamiliar  with  emergency 
management, 

•  Cushion  small  contributors  from  government  bureaucracy  and  procedures  that  they  do  not  have 
the  administrative  depth  to  manage,  and 

•  Provide  government  agencies  with  a  single  point  of  contact,  experienced  project  management, 
and  confidence  that  they  will  apply  quality  control  and  deliver  an  effective  product  on  time. 

The  organization  providing  that  cohesion  could  be  an  academic  center  of  excellence,  a  non-profit 
company,  a  national  laboratory  or  FFRDC,  or  a  council  of  interested  project  managers  within  FEMA  or 
DHS  -  a  kind  of  “think  tank”  model.  The  organization  could  conduct  some  basic  research  to  advance  the 
state  of  the  art,  as  well  as  provide  a  direct  service  in  managing  projects  to  create  actual  training  materials. 
Ultimately,  the  change  (to  leveraging  serious  games)  must  come  from  within  FEMA/DHS,  but  the 
organization  supporting  and  enabling  that  change  might  be  inside  or  outside  the  government.  <Source: 
Industry,  Academia,  Government,  National  Laboratory> 
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It  is  common  for  an  entertainment  game  that  is  ultimately  successful  to  receive  investor  funding 
when  it  is  still  very  rough  -  e.g.,  only  75%  of  a  good  design,  plus  needing  development  and  polish. 
Investors  and  publishers  have  developed  skill  at  identifying  games  that  are  promising  and  ready  to  be 
brought  to  completion.  That  model  would  map  quite  well  to  a  government  sponsorship  model  in  which  a 
national  laboratory  or  academic  center  of  excellence  designs  a  game  that  targets  appropriate  skills  and 
learning  objectives,  then  passes  it  off  to  a  commercial  company  to  mature  and  complete  the  game. 
Companies  like  MassDigi  and  consortiums  of  independent  developers  could  find  game  developers  to 
finish  prototypes  developed  by,  say,  national  laboratories  to  target  particular  skills.  <Source:  Industry> 

Consider  a  project  structure  in  the  style  of  a  DARPA  challenge.  In  that  model,  the  job  of  the 
academics,  national  laboratories,  and  nonprofits  would  be  to  carefully  define  the  soft  skills  to  be  targeted, 
but  then  leave  a  lot  of  freedom  for  how  bidders  meet  those  goals.  Participants  would  receive  a  reward  for 
winning  but  keep  their  IP,  giving  additional  incentive  for  top  performers  to  participate.  This  model  would 
get  people  looking  into  those  areas  and  potentially  seed  games  that  serve  the  role  desired,  with  more 
creative  freedom  and  a  higher  chance  of  creating  a  breakthrough  technique.  <Source:  Government, 
National  Laboratory> 

Going  through  the  normal  RFP  process  can  have  a  high  (50%+)  rate  of  failure  on  projects  and  could 
be  even  worse  for  newer  approaches,  such  as  game-based  training.  It  might  be  better  to  instead  focus  on 
an  industry-owned  and  government-influenced  game.  The  government  could  provide  loose  guidance  on 
the  context  and  data  to  be  collected,  but  end  up  with  better  graphics,  sound,  playability,  and  engagement. 
<Source:  Government,  Industry> 

Whatever  process  is  used,  the  sponsor  needs  to  have  a  healthy  acceptance  of  (and  support  of)  risk 
and  failure.  Pursuing  novel  techniques  always  involves  trial  and  error  and  lots  of  iteration,  and  giving 
performers  “freedom  to  fail”  is  widely  regarded  as  a  core  element  of  fostering  innovation  and 
modernizing  a  field.  Sponsoring  a  larger  number  of  projects  at  lower  budget  levels  and  with  greater 
freedom  given  to  the  performers  can  provide  better  aggregate  results  and  reliability  than  heavy-handed, 
large-budget,  sole-source  programs.  <Source:  Industry,  Government 
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APPENDIX  B.  PRELIMINARY  OPTIONS  FOR  STRUCTURING  A  GAME 
DESIGN  COMPETITION  (MODEL  B) 


This  appendix  explores  two  options  for  how  to  build  a  collaboration  between  government  training 
organizations  (especially  those  in  DHS  and  FEMA  dealing  with  emergency  response)  and  the 
independent  video  game  community  via  competitions.  It  proposes  some  structures  for  organizing  and 
motivating  such  a  collaboration  on  both  sides  of  the  fence.  This  document  is  meant  to  be  a  discussion 
point  for  getting  feedback  from  both  communities  on  the  types  of  contests,  rewards,  and  restrictions  that 
would  be  appealing  and  acceptable.  It  is  not  intended  to  address  the  details  of  the  overall  project  logistics, 
execution  costs  (other  than  the  actual  prize  money),  or  legal  phraseology. 

Serious  games  have  demonstrated  the  ability  to  target  important  skills  that  are  otherwise  very  hard 
to  teach,  evaluate,  or  study.  However,  the  experience  of  the  serious  games  community  has  been  that  it  is 
hard  to  reliably  create  effective  games  -  be  they  serious  game  or  entertainment  games  -  and  the  best 
approach  is  often  to  cast  a  wide  net.  Some  of  the  most  effective  new  ideas  come  from  young  minds  and 
unknown  rising  stars.  Those  are  the  kinds  of  people  who  can  capture  critical  learning  goals  in  a  game 
while  also  making  it  engaging  and  immersive,  and  do  so  with  new  ideas.  However,  those  are  also  the 
people  who  are  typically  not  targeted  by  traditional  government  sponsorship,  since  they  do  not  have  the 
experience  or  corporate  infrastructure  to  pursue  government  funding. 

Our  goal  is  to  bring  together  these  two  communities  -  government  agencies  interested  in 
modernized  training  via  serious  games  and  independent  game  developers  with  talent  and  fresh  ideas.  We 
want  to  do  so  with  a  mechanism  that  will  be  familiar  to  both  sides.  For  the  government  side,  this  means 
having  an  experienced  national  laboratory  oversee  the  project  and  provide  the  kind  of  guidance  and 
reporting  that  is  tried  and  true  in  government  projects.  For  the  independent  developer  side,  this  means 
accessing  a  large  number  of  developers,  letting  them  experiment  without  the  overhead  of  dealing  with 
government  directly,  and  providing  them  loose  guidance  on  the  kinds  of  products  we  would  like  to  see. 

The  solution  is  to  do  a  contest  or  “game  jam”  with  cash  prizes,  supported  by  a  development  toolkit, 
and  supervised  by  researchers  with  experience  in  games,  training,  and  emergency  response.  Tapping  the 
independent  community  accesses  a  talented  and  often  underpaid  community  of  creative  thinkers  and 
software  developers.  The  key  insight  is  that  relatively  small  amounts  of  prize  money  provided  to  a  wide 
range  of  participants  can  generate  a  large  number  of  new  ideas  at  relatively  low  cost.  Even  if  only  a  few 
of  those  ideas  are  any  good,  the  end  result  will  be  higher-quality  designs  at  a  lower  dollar  cost  than 
putting  all  the  eggs  in  one  basket  up  front  with  a  single  design  and  a  single  developer.  Once  this  process 
has  revealed  the  gems,  they  can  be  pursued  in  a  traditional  sponsored  fashion  with  a  much  higher  chance 
of  project  success  and  higher  ultimate  positive  impact.  This  document  addresses  the  question  of  how  to 
motivate  participation  and  structure  the  process  of  finding  a  wide  range  of  ideas,  down  selecting  those 
ideas  to  a  solid  core,  and  developing  that  core  to  a  point  ready  for  traditional  sponsorship. 

Reaching  the  independent  game  community  is  easy.  A  booth  at  the  annual  Game  Developer’s 
Conference  (GDC)  or  smaller  gatherings  such  as  the  annual  Boston  Festival  of  Independent  Games 
(BFIG),  are  inexpensive,  effective  means  of  getting  the  word  out. 
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PITCH  TO  INDEPENDENT  DEVELOPERS 


The  following  pages  outline  thoughts  on  how  to  present  the  competition  to  the  game  development 
community.  We  have  picked  a  rough  estimate  of  a  prize  pool  of  $100k.  A  pool  half  that  size  would  still 
be  considered  reasonable  for  a  high-profile  game  competition,  but  the  $100k  level  would  stand  out  and 
gather  a  lot  of  attention  in  the  community.  These  numbers  are  intended  to  give  a  concrete  idea  of  what  the 
prize  structure  would  look  like  to  participants,  and  what  the  total  cost  might  be  to  a  sponsoring  agency. 
They  are  not  intended  to  be  recommendations,  careful  predictions,  or  to  account  for  all  costs  associated 
with  the  project. 
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WANTED:  Independent  game  developers  and  designers  to  help  train  emergency  responders 

•  Sharpen  your  game  creation  skills 

•  Network  with  other  designers  and  developers 

•  Earn  some  money 

•  Have  a  chance  to  earn  a  lot  of  money 

•  Create  open-source  software  to  help  the  community 

•  Make  a  positive  impact  on  the  world 

•  Have  fun  helping  a  humanitarian  cause 

We  are  seeking  game  designers  and  developers  from  companies  of  all  sizes  (or  from  no  company  at 
all!)  to  help  create  serious  games  to  support  emergency  response.  We  want  to  see  new  and  creative  ideas 
for  leveraging  games  to  teach,  evaluate,  and  study  effective  emergency  response.  We  believe  that 
effective  serious  games  need  to  be  both  targeted  and  engaging,  and  we  think  that  the  best  people  to  build 
such  games  are  small  companies  and  independent  developers  with  fresh  ideas  and  experience  embedding 
strategic  and  social  dynamics  into  games  in  a  way  that  is  fun. 

The  game  you  design  can  be  from  the  point  of  view  of  first  responders,  emergency  managers, 
survivors,  adversaries  causing  the  disaster,  or  any  other  perspective  that  you  think  will  accomplish  the 
target  goals  and  be  engaging  to  users.  It  does  not  have  to  be  completely  serious  as  long  as  it  targets  the 
appropriate  skills.  It  is  perfectly  fine  for  zombies  and  elves  to  show  up  as  part  of  creating  situations  that 
will  target  appropriate  skills;  but  if  you  can  make  the  game  fun  without  zombies,  that  is  good  too. 

All  participants  will  receive  information  about  the  kinds  of  skills,  dynamics,  and  data  collection  we 
are  looking  for  in  these  games,  and  then  will  be  given  free  rein  to  try  out  novel  approaches.  You  will  have 
access  to  the  EM  Dev  Kit,  providing  resources  to  help  you  build  games  about  disasters,  including  physics 
models,  population  models,  and  relevant  art  assets  for  common  development  platforms. 

The  best  games  will  be  selected  for  continued  development;  candidates  will  be  offered  a  contract  to 
continue  development  in  collaboration  with  researchers,  training  professionals,  and  real  first  responders. 

Accepting  the  contract  is  not  mandatory  to  receive  the  prize  money  for  earlier  rounds,  and  a  follow- 
on  contract  is  not  guaranteed.  If  you  show  you  can  make  a  creative  and  solid  product,  there  are  many 
government  agencies  looking  to  sponsor  work  in  that  area. 

Participating  teams  must  meet  minimum  quality  standards  to  receive  rewards,  at  the  discretion  of 
the  judges.  Judges  will  include  emergency  response  professionals,  researchers  from  a  national  laboratory, 
and  other  game  designers  and  developers  like  you. 

All  source  code,  game  concepts,  and  art  assets  submitted  to  the  contest  will  be  open  sourced.  They 
will  therefore  be  available  for  both  government  use  and  private  use  going  forward. 
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OPTION  I:  MULTIPLE  QUALIFIER  CONTEST 
Round  1:  Apply  for  participation 

•  1  month  timeline  from  announcement  of  topic 

•  No  prize  money  at  stake  yet 

•  40  teams  qualify  to  continue  to  Round  2.  They  may  recruit  additional  members  for  later  rounds 

•  A  template,  list  of  goals,  and  sample  games  will  be  provided  to  guide  the  applications 
Round  2:  Build  a  quick  prototype 

•  1  month  timeline 

•  Submissions  are  graded  by  the  judges  according  to  their  training  potential,  creative  novelty,  and 
polish 

-  Grade  A:  $5,000;  outstanding  product;  up  to  3  submissions  can  be  As 

-  Grade  B:  $1,500;  solid  product;  up  to  20  submissions  can  be  B  or  better 

-  Grade  C:  $500;  meets  minimum  standards 

-  Grade  D:  $0;  does  not  meet  minimum  standards 

•  Max  total  prize  pool  =  $46,500 

•  Up  to  5  games  rated  A  or  B  will  move  on  to  the  next  round 
Round  3:  Refine  your  prototype  with  input  from  professionals 

•  1  month  timeline 

•  Each  design  meeting  expected  quality  standards:  $10,000 

•  Best  design  will  also  be  a  candidate  for  a  development  contract  to  complete  the  game  -  there  is 
no  guarantee  of  getting  a  contract  even  if  you  win,  but  you  will  retain  credit  for  your  design  and 
all  prize  money  won  in  rounds  1  through  3 

•  Max  total  prize  pool  =  $50,000. 

Round  4:  Finalize  the  game  to  be  easy  to  play,  easy  to  learn,  and  easy  to  extend  to  new  scenarios 

•  6-12  month  timeline  depending  on  the  sophistication  of  the  game 

•  Judges  will  decide  if  winners  of  Round  3  will  be  given  a  development  contract  or  serve  as 
consultants  for  a  separate  company  contracted  to  finalize  the  game 
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•  If  hiring  the  winning  team,  they  would  probably  be  paid  $100k-$500k  depending  on  the  type  of 
game  created 

•  If  not  hiring  the  winning  team,  the  sponsoring  agency  would  pursue  a  traditional  single¬ 
performer  contractor  model,  with  the  winning  team  retaining  credit  for  the  design  and  an 
opportunity  to  consult  on  the  evolution  of  their  idea  -  the  contracted  company  would  have  the 
final  say  on  all  design  decisions,  but  would  be  encouraged  to  take  input  from  the  contest 
winners 

•  This  phase  involved  making  the  game  easily  extensible  to  handle  new  scenarios,  so  that  there 
would  be  a  very  little  cost  to  adapt  it  to  integrate  with  other  courses  with  similar  goals 
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OPTION  II:  GAME  JAM 


Same  as  Option  I,  except  Round  2  is  changed  as  follows: 

Round  2:  Game  jam  at  a  local  venue 

•  2-3  day  timeline,  occurring  over  a  single  weekend 

•  Food  will  be  provided  for  all  participants,  who  will  work  on  their  games  in  a  single  room 

•  Submissions  are  graded  by  the  judges  according  to  their  training  potential,  creative  novelty,  and 

polish 

-  Grade  A:  outstanding  product;  up  to  5  games  can  be  As 

-  Grade  B:  solid  product;  up  to  10  games  can  be  B  or  better 

-  Grade  C:  meets  minimum  standards 

-  Grade  D:  does  not  meet  minimum  standards 

•  Every  team  will  receive  a  reward  that  scales  up  based  on  the  total  number  of  good  games  that 

emerge 

-  Each  A  game:  +$100  to  each  team 

-  Each  B  game:  +$25  to  each  team 

-  Each  C  game:  +$10  to  each  team 

-  Each  D  game:  +$0  to  each  team 

•  Max  total  prize  pool  =  $38,000 

•  Up  to  5  A  or  B  grade  games  will  be  selected  to  move  on  to  Round  3;  those  teams  will  move  on 

to  the  next  round,  but  all  teams  will  get  a  prize  boost  because  of  their  success 

Note  that  the  culture  of  game  jams  is  usually  one  of  collaboration,  not  competition.  Many 
developers  participate  in  order  to  network  and  learn  from  others,  not  to  prove  their  superiority.  As  such, 
the  prize  structure  for  Option  II  encourages  collaboration,  but  still  gives  credit  to  teams  who  produce 
good  products. 

One  big  benefit  of  a  game  jam  is  that  the  collaborative  nature  allows  for  more  cross-fertilization  of 
ideas.  A  team  whose  game  is  not  coming  together  has  a  social  and  financial  incentive  to  help  other  teams 
succeed,  improving  the  overall  quality  of  the  final  games.  The  main  drawback  of  a  game  jam  is  that  the 
short  time-frame  limits  the  resulting  products  to  the  talent  in  the  room.  With  Option  I,  a  participant  might 
recruit  someone  to  help  fill  a  gap  on  their  team,  whereas  there  is  no  opportunity  to  do  so  with  Option  II. 
Both  options  can  be  effective  for  the  right  type  of  game.  Option  I  is  more  likely  to  produce  polished 
products,  but  will  have  less  originality  and  brainstorming  potential.  Option  II  is  more  likely  to  produce  a 
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new  or  clever  solution,  but  the  products  will  require  more  polish.  With  both  options,  the  subsequent 
refinement  stages  (rounds  3  and  4)  will  be  important  to  get  the  product  ready  for  use. 

Both  of  these  options  have  been  reviewed  with  independent  game  developers  who  are  active  in  the 
community.  They  helped  calibrate  the  prize  structures  to  be  “generous  but  not  too  good  to  be  true”  and 
define  the  structure  to  mesh  with  current  “indie  culture”  while  also  maximizing  the  number  of  good 
games  that  emerge.  Opinions  were  split  over  which  option  was  more  desirable,  but  all  interviewees 
thought  that  both  options  would  be  effective. 
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GLOSSARY  OF  TERMS  AND  ACRONYMS 


AI 

BAA 

BFIG 

CBRNE 

DARPA 

DHS 

DHS S&T 

DoD 

FEMA 

FFRDC 

game  jam 

GDC 

GOTS 

IMAT 

IND 

indie 

IP 

MMO 

MOBA 

modding 

NUSTL 

RFP 

RTS 

SME 


artificial  intelligence 

broad  agency  announcement 

Boston  Festival  of  Indie  Games 

Chemical  Biological  Radiological  Nuclear  Explosive 

Defense  Advanced  Research  Projects  Agency 

Department  of  Homeland  Security 

DHS  Science  and  Technology  Directorate 

Department  of  Defense 

Federal  Emergency  Management  Agency 

federally  funded  research  and  development  center 

a  short  ( 2-4  day)  prototype-building  competition  for  game  developers 

Game  Developer’s  Conference 

government  off-the-shelf 

Incident  Management  Assistance  Team 

Improvised  Nuclear  Device 

independent  software  developer,  either  freelance  or  part  of  a  very  small  company 
intellectual  property 

massively  multiplayer  online  (a  type  of  game) 
multiplayer  online  battle  arena  (a  type  of  game) 

when  a  fan  modifies  a  game  after  its  release  with  sanctioning  by  the  publisher 

National  Urban  Security  Technology  Laboratory 

Request  for  Proposals 

real-time  strategy  (a  type  of  game) 

subject  matter  expert 
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